MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Summary

The manuscript presents a detailed study on the development and application of MiniCPM, a series of small language models (SLMs), which demonstrate comparable capabilities to much larger language models (LLMs) in the range of 7B-13B parameters. The authors have addressed a timely topic in the machine learning community, focusing on resource efficiency and practical utility of SLMs.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

1) The paper is well-motivated, addressing both the economic and practical challenges of deploying large models. 2) The MiniCPM models are robustly presented, with detailed descriptions of the architecture, training methods, and innovative approaches like the WSD learning rate scheduler. 3) The experimental section is comprehensive, demonstrating the models' performance across various benchmarks, which substantiates the authors' claims.

Reasons to reject

1) Does the emergence pattern of Small Language Models (SLMs) mirror that of Large Language Models (LLMs)? If there are differences, can the insights gained from SLMs still be applicable for the development of LLMs? 2) The recent advancements with models like Llama-3 suggest that smaller-scale models can achieve enhanced performance with increased training data. Does this paper offer any additional insights or conclusions in this regard?

Questions to authors

1) What is the total model size when embedding parameters are included? 2) The manuscript lacks a discussion on the utilization of hardware resources, such as GPUs. Could you provide details on this aspect?

Summary

This paper investigates the capabilities of small language models (approximately 2 billion parameters), a critical area in current language model research. It introduces MiniCPM, a family of models with 1.2B and 2.4B parameters that exhibit competitive performance. This success is attributed to a thorough hyper-parameter scaling analysis on smaller-scale models and a novel training strategy employing a learning rate scheduler named Warmup-Stable-Decay. While the paper is generally well-written and provides detailed empirical results and discussions, some issues with notation clarity and figure presentation persist, and additional ablation studies are recommended to validate the effectiveness of the proposed methods.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- Exploring the capabilities of small language models is significantly valuable for advancing research in both small and large language models. - The paper presents a comprehensive and effective training strategy for small language models, alongside clear decision-making insights derived from empirical results. These contributions are likely to inspire further research in the field. - The proposed model family shows impressive performance across a variety of benchmarks, even when compared to much larger models.

Reasons to reject

- The paper contains ambiguities and possible errors in its notation, which impede understanding. For instance: - This paper uses notations like "WSD(80N,8N)'' multiple times. However, there is no definition about what the two items in parentheses represent. $WSD(T,D)$ in Appendix B.1 is also not defined. - In the "Decay Stage" section, the equation for exponential annealing $f(s-T)=\eta \times 0.5^{(s-S)/T}$ appears to be incorrect. The variable $\eta$ seems out of place, and when $s=T$, the learning rate should equal $\eta$. - Several figures are poorly presented: - The labels in Figures 1 and 2 are too small to read. - It is difficult to distinguish between the colors of different lines in Figures 5 and 6. - Despite descriptions suggesting the presence of a curve showing the 0.036B model (with 4 times compute) matching the performance of the 0.17B model with WSD (40N, 4N), this curve is absent from Figure 6. - This paper highlights the benefits of introducing high-quality SFT data at the beginning of annealing. However, mixing SFT data with pretraining data after the pretraining could also improve the performance. That is to say, what may be important is the data mixing operation rather than the training stage. The ablation study did not shows whether mixing SFT data at the beginning of annealing is the best choice. - Most baselines in Table 2 do not use SFT, whereas MiniCPM is trained with both pretraining and SFT data. Additional discussion or experiments are needed to determine how much of MiniCPM’s success is due to the inclusion of SFT data.

Questions to authors

- Are optimal batch size and optimal learning rate independent? Analyses in Section 3.2 and 3.3 seem to be done separately. - Will the training code, including the implementation of WSD, be made publicly available to facilitate reproducibility in research?

Summary

In this paper, the authors study the training dynamics of small language models (i.e., models ranging from 1.2-2.4 billion parameters). Specifically, they focus on what they call “model wind tunnel” experiments. This involves training and studying model developing in 3 parts: 1) hyper-parameter selection (mostly focusing on network width and depth scaling parameters); 2) optimal batch sizing and; 3) learning rate scheduling. The most novel part of their work seems to be in the last part related to the development of what they call “warmup stable decay” (WSD), their learning rating scheduler that is conducive to continuous training and training without a fixed token length. These experiments above lead to the following results (centering around experiments on C4): a new empirical bound on the optimal relationship between batch size and C4 loss that helps to minimize both token amount and loss (reported at the end of Section 3.2). Compelling empirical results on the efficacy of their WSD scheduling method (which breaks the scheduler into 3 stages: warmup, stable and decay stages, informed by systematic experiments they performed on cosine-warmup and its sensitivity to various step parameters); in particular, they report impressive losses with very small models on C4 (Figures 5 and 6). Based on these insights, they then train a family of small language models called miniCPM. Their training pipeline is comprehensive, ranging from full pre-training (at the scale of trillion of tokens using aggregations of multilingual publicly available pre-training corpora), to supervised instruction tuning and DPO-style alignment. They also tested across different architectures (e.g., MoE derivates and long-context versions of their models). The main novelty in this pre-training setting is the introduction of SFT data into the decay stage of their pre-training and scheduler (while I know this has become popular d recently, I worry that this makes their models have an unfair advantage over some of the other models they are comparing against; more about this below). The results are impressive and, after looking through the details in the appendix, I was happy to see that not only were all the model comparisons produced by the authors themselves using a single tool, but that performance reported is based on each model’s best performance either through perplexity-based evaluation or direct generation. While I initially found Table 2 to be a bit confusing (highlighting would be very helpful here!), I was particularly struck by the relative average performance of 13-40B models against MiniCPM-2.4B.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

- A very careful study on optimal model training strategies for small LLMs, which is an important area of research that, as the authors convincingly argue, touches on important issues related to practical deployment of LMs. - The paper is very easy to read and I found all the experiments to be well motivated. I believe that their experiments will motivate further research in this area. - A new suite of small LMs that I think will be used and embraced by the community (the question of whether these models will be released was not addressed; see questions below).

Reasons to reject

- My main concern centers around the introduction of instruction data into the pre-training stage, and whether this makes the comparisons fair to many of the models in Table 2 (models that presumably didn't do this). While I don't think that this is grounds for a rejection, I would like to see the authors motivate and address this. If there were any ablations on this (maybe that I would missed), it would be helpful to report. (Given that this was done in the most recent OLMO 1.7 model, this might be worthwhile to compare against)

Questions to authors

- Will you models be released, and if so under what license? - I was surprised not to see (unless I missed it) details about the the particular optimizer you used. Can you clarify this? - I was surprised to that your SFT stage includes training on 6 billion tokens. Did you report what sources you used? - you called the alignment stage RLHF, but as far as I can tell, you weren't using RL, because you were training with DPO, right? Please clarify. other comments - The caption is Table 1 is very hard to read, please revise.

Summary

This work presents a suite of models called the MiniCPM models, which are language models of moderate size (~1.2B and ~2.4B parameters) that perform comparably to some of the strongest 7B models that are available (incl. Mistral-7B). Their main contribution is introducing a warmup-stable-decay (WSD) learning rate scheduler and analyze properties of this. They integrate WSD into their system and enjoy performance bumps. Finally they contribute an MoE model and long-context model based on the MiniCPM architecture.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

1) WSD seems like quite a simple approach and can be integrated fairly easily into LM pretraining. 2) Performance seems good in general, being on par with or close to models 3-4x the size. 3) Analysis of the hyperparameters to get something that ends up working is good. 4) Releasing a suite of models including an MoE, long context, and potentially a vision one (this was stated to maybe be a part of an upcoming paper, but also slated in here, which confused me).

Reasons to reject

1) Although it seems like WSD works well on very small scales that you've tested, it would've been nicer to see how the exact same model with the exact same data order without WSD would do across the board. I think this could be computationally prohibitive, but that baseline would be nice to see. 2) It seems like nearly all of the contribution here is the algorithm for WSD, but that's conflated with the fact that you're releasing a set of performant models, so its a little distracting and poorly organized. (minor point) 3) There are some clarity issues that I expand a little bit more about in the questions to the authors. (minor point)

Questions to authors

Clarity things 1) N throughout is number of parameters of the model right? When you say S = 20N for a 1B model this would be 20B tokens of training, right? It would be great to make this more clear in the body of the text. 2) The paper was hard to follow and there are a lot of awkward and/or redundant phrasing that made things unclear (e.g. in the batch size section 'the same number of GPUs is kept constant').

Reference updating

[3] was updated into a paper very recently: Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations https://arxiv.org/pdf/2405.18392

Response to the rebuttal

Thanks for answering the questions and concerns I have. I think I under-valued the demonstrated utility of WSD, so after considering some of these things I'm inclined to raise my score to a 7. I'm updating the review to reflect this.