Reusing Pretrained Models by Multi-linear Operators for Efficient Training

Summary

The authors propose a new approach, Mango, to use a small model as initializer to larger models in order to speed up the training of the larger model. Mango proposes to learn a mapping of weights from the smaller model to the larger model, not layer by layer or other restricted form of mapping, but a full mapping of weights. Because the matrix that would represent this linear mapping of weights being huge (in the order of M x N where M and N are respectively the number of parameters of the small and the large model), Mango proposes a particular form of low rank factorization, allowing to keep a full mapping but constraining each of its sub-mapping to rank 1. The authors show that this mapping can be learned in as few as 100 steps, making its computing cost negligible. By doing so, Mango shows that full mappings are useful, and shows faster convergence than the other methods on 3 types of architectures : vision transformers, BERT models and GPT models.

Strengths

* Well written paper, with very good motivating examples * Good theoretical part, with the need of low rank factorization clearly introduced * Compelling results of 3 different models: Vision Transformers, BERT and GPT * The gaps in the results are significant: on vision transformers, Mango converge in 2/3 of the time of the next best method, and in 1/4 of the time of naive from scratch training. The savings on BERT are less important but still quite significant.

Weaknesses

The results are constrained to the case of Small to Base size transformers mapping : * Deit-S to Deit-B * BERT-S to BERT-M * GPT-Base from GPT-Small The most interesting regime for this class of method is with bigger transformers since they are the most power hungry. The paper would gain a lot of impact by showing that Mango works for larger models (for instance to initialize a ViT-G from a ViT-H). I would definitely increase my rating if such a proof was shown, especially with a low rank such as 1. Another interesting result would be to know, as we deal with larger models, whether it is best to initialize them with the larger pre-trained available, or if we can initialize them with Mango from a small model. For example, to initialize a ViT-G, should we preferentially use a ViT-H or ViT-L ? And how much would the difference be ?

Questions

* The authors mention the CO2 consumption of training large models. Do you have an estimate of those emissions and the type of reduction achieved by Mango? Is the improvement proportional to the improvement in FLOPs?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper addresses the issue of high resource costs in training large AI models from scratch. To this end, the authors propose a novel approach that linearly correlates each weight of the larger target model to all weights of the smaller pre-trained model, employing multi-linear operators to decrease complexity and manage resource demands. Experimental results on ImageNet reveal that the proposed method can significantly reduce training costs while maintaining performance.

Strengths

The main idea is well-written and easy to follow. The introduced Mango Operator is both innovative and intuitive, exhibiting a remarkable ability to reduce training costs.

Weaknesses

The experimental section lacks essential details, particularly around how the acceleration ratio is measured. Also, there are potentially contradictory statements about training steps and epochs, and the use of only theoretical metrics such as FLOPs for measuring training cost.

Questions

1. In the proposed method, the authors first concatenate weights across layers. However, given that the row count of $\mathbf{W}_j^{OUT} \in kI \times O$ differs from that of other weight matrices, how is this discrepancy addressed? Elaborating on this point would enhance the clarity of the method. 2. In Equation (5), the authors suggest decomposing $\mathbf{S}$ using a Tensor Ring Matrix Product Operator (TP-MPO) into four smaller tensors. I'm curious to know if it would be possible to decompose $\mathbf{S}$ into five or six smaller tensors. 3. Essential details seem to be lacking in the experimental section. Could the authors clarify how the acceleration ratio was measured? It would be better to provide more descriptions, which would strengthen the paper. 4. There are a few statements in the experiments section that cause confusion. The authors mention that the Mango operators are trained for 100 steps, which is claimed to take negligible time. However, they also state that the model is trained for 300 epochs. Could the authors clarify these seemingly contradictory statements? 5. In the experiments, the authors measure the training cost using FLOPs, which is a theoretical metric. It might be more insightful if the authors could also offer measurements related to GPU training hours, providing a more practical perspective on computational cost. 6. The proposed method significantly reduces training costs compared to the traditional 'training from scratch' approach, primarily by leveraging the knowledge of the pre-trained model. I am intrigued to understand if this approach would sustain its comparable performance if we were to remove the training cost restriction and instead train until convergence.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have provided a dedicated discussion on the potential limitations of their work.

Summary

This paper proposes a new method for reusing pre-trained models to a larger target model. They propose mango which decomposes the parameters into 4 smaller matrices and then learns to grow these parameters. They show their method requires less FLOPS to achieve as good as performance other baselines across vision and NLP tasks.

Strengths

The paper was easy to follow and the experiments are strong across various domains. Mango seems well motivated in allowing for interactions between all weights in the model.

Weaknesses

Missing background on tensor ring matrix product - since this is the main method proposed Assuming the acceleration ratio is in terms of FLOPS, then there are no experiments showing the wall-clock time of Mango. Decreasing FLOPS is only useful if it decreases wall-clock time (assuming the same machine). The models seem to be small, only growing from small to base variants, not base to large variants

Questions

Is the acceleration in terms of FLOPS or wall clock time? How does the multiring operator allow for interactions between all weights in the model? How does S_L capture the relationship between layers?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

See weaknesses above.

Summary

This paper studied the problem of solving a proper initialization for large deep learning models on top of smaller pre-trained networks. The key idea is to concatenate the weights of pre-trained models to a large matrix, and map the matrix to the parameter space of the larger model to be trained with a multi-linear operator. In this procedure, the rank of the transformation matrix can be manually adjusted. Experimental results on NLP/CV-based Transformer models are provided.

Strengths

1. The problem of efficient training is practically important. 2. The proposed method is well-motivated. 3. The design of the multi-linear operator is interesting. 4. Mango can be applied to both NLP and CV models.

Weaknesses

1. The writing needs to be improved. It is difficult to understand Eqs. (4-5), which are the key contributions of this paper. 2. Currently, the topology of the network is not considered in Mango (e.g., the depth of the layer corresponding to each weight matrix). I think it may be beneficial to leverage this characteristic in Mango. 3. Can we introduce non-linear operation to Mango to improve training efficiency? 4. In Figure 7, the comparison with training from scratch may be unfair. Take DeiT for example. Obtaining DeiT-S needs a notable amount of computation. Hence, the computational cost of Mango should not start from zero FLOPs. 5. This basic baseline may be interesting: initializing the large model (e.g., DeiT-B) by pre-training it using the same amount of computation as training the smaller model (e.g., DeiT-S). 6. Can Mango improve the final accuracy of the model? 7. In the field of CV, it would be interesting to validate the effectiveness of Mango on top of more network architectures, e.g., Swin Transformer. 8. Some related works should be discussed or compared with Mango: - Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., & Yang, Y. (2022). Automated progressive learning for efficient training of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12486-12496). - Budgeted Training for Vision Transformer, ICLR'23 - Wang, Y., Yue, Y., Lu, R., Liu, T., Zhong, Z., Song, S., & Huang, G. (2022). Efficienttrain: Exploring generalized curriculum learning for training visual backbones. arXiv preprint arXiv:2211.09703.

Questions

Please see the weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Personally, I think this paper may not have potential negative societal impacts.

Summary

The contribution of this paper lies in proposing a new method to accelerate the training of large models by leveraging the correlations among all weights in the model, instead of just mapping partial weights as in existing methods. The authors' approach utilizes multi-linear operators to reduce computational and spatial complexity, enabling acceptable resource requirements. Experimental results demonstrate that their method has excellent performance in accelerating training, outperforming existing methods.

Strengths

+ The method proposed in the paper can leverage the potential correlations among all weights in the model, significantly reducing computational costs. + the authors conducted experiments in both image and natural language processing domains and achieved results that surpass existing methods. This indicates that their proposed approach has a certain level of generality and applicability, and can significantly accelerate model training across different domains.

Weaknesses

+ In the experiments of the paper, only DeiT-B, BERT-Base, and GPT-Base were used, and no further exploration was done on larger models. + The performance of Mango, Ligo, and bert2BERT was improved in natural language processing methods, while only marginal gains were achieved in image tasks, and the experiment lacked analysis and explanations. + For Figure 6(b), there is continuous improvement in accuracy with increasing rank, but there is a lack of experimental comparison for larger rank values.

Questions

+ In line 196 of the paper, for the loss function in Equation (6), what is the specific design of this loss function for the operators? Is it task-specific or fixed?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

It might be interesting if the results can be extended to large language models.

Response to Authors

Dear Authors, Thanks for the reviewers' rebuttal, which has solved all of my concerns. Warm regards, Reviewer 9N2L

Post-rebuttal comments

I highly appreciate the efforts of the authors in rebuttal. Most of my concerns have been addressed. I'm happy to keep my positive rating (i.e., Borderline Accept).