OLMoE: Open Mixture-of-Experts Language Models

Thank you for your detailed review and we are glad you find the analysis section informative — We indeed hope that this lowers the barriers of entry to MoEs and helps the growth of our community. **Regarding the model size being smaller than some other MoEs:** We fully agree with your point and also mention it in our limitations section (Appx K). Larger models are more resource-intensive - we hope that by fully openly releasing OLMoE, we enable the community to build on it and scale up even further in the future (something we are also currently investigating!). **Regarding the necessity of expert parallelism:** Great question — As you mention, we do not need expert parallelism at these small scales but it may be required at larger scales. The DBRX model with 36B active and 132B total parameters by Databricks [1] was also trained with dropless routing similar to OLMoE, so we should be able to scale up to that size at least without having to drop tokens. Another avenue could be switching to expert choice routing which does not drop tokens regardless of the scale - While we found it to perform worse in the paper, we hypothesize that it may perform better in multimodal settings. We have added a note on expert parallelism in Appendix K and updated the paper, thanks again for raising this. [1] Databricks. Dbrx, 2024. URL https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm.

Thank you for your review and highlighting the strong results and reproducibility aspects. Regarding your Weaknesses: **No other models available with the same configuration:** Thanks for bringing this up; indeed, we could not find state-of-the-art MoE models matching our configuration. One of the motivating factors for our work is the lack of open models with this combination of active/total parameters. Specifically, we chose this configuration with three reasons in mind: **1. Model training:** 7B total parameters allow for full-parameter training on a single GPU. Specifically, our model can be trained on one 80GB VRAM GPU (e.g. A100 or H100) as it requires around 70GB of memory for [training the model in 16-bit with an 8-bit optimizer](https://blog.eleuther.ai/transformer-math/). This makes the model significantly more accessible to researchers who are often constrained by a single GPU and also bypasses the need for more complicated distributed training across multiple GPUs. A slightly larger model (e.g. JetMoE-2B-9B [1]) may no longer fit under this setup. **2. Model usage on laptops:** Laptops commonly have around 16GB of RAM, thus 7B parameters corresponding to 14GB in 16-bit precision perfectly fit into most laptop's RAM. With the speed of 1B parameters, the configuration of OLMoE-1B-7B could make it an ideal local assistant. **3. Model usage on phones:** We have been able to run OLMoE-1B-7B on an iPhone by quantizing the model to 4-bit after which it requires around 3.5GB. This is just below the ~5GB RAM limit that is [commonly imposed for an iOS app](https://github.com/thebaselab/codeapp/issues/25) leaving ~1.5GB of RAM for other functionalities of the app. Thanks to the 1B active params OLMoE-1B-7B runs very fast on smartphones; we were able to run it at 110 tokens/second on an iPhone 16. This enables applications that might not make sense with larger and slower models, such as having the model quickly read multiple long files and summarize them. To compensate for the lack of a model with a similar configuration, we compare with larger models like JetMoE (2B/9B) [1] and Qwen (3B/14B) [2] in the paper, state-of-the-art models for their respective parameter configurations. We find that OLMoE is competitive with these larger models, thus we would expect OLMoE to significantly outperform a similar model to theirs with a 1B/7B configuration. We have added this discussion on the 1B/7B configuration to the paper in a new Appendix section (J). We thank the reviewer for providing an opportunity to reflect on this important motivating aspect of our work! **Design choices based on prior work:** Thanks for highlighting this – our work heavily builds on important prior work in the field. The two areas of novelties that you mention include many new findings such as for “design space exploration”: the stronger performance of dropless token choice [3] over expert choice [4] and the limits of sparse upcycling [5] among others; and for “routing”: new measures for analysis including expert co-activation, specialization metrics, and router saturation. In addition to these two areas, we think that there are two more with novelties: **1. Some design choices themselves and their combination:** Some of our design choices are new (e.g. our use of a truncated normal initialization for an MoE; the specific dataset we mix), as well as some combinations are new (e.g. using both finegrained experts and router z-loss in one model). Overall, this leads to a novel and reproducible recipe for training a state-of-the-art MoE model. **2. Openness:** All aspects of OLMoE being fully open-source including 244 checkpoints is also a first among MoEs to the best of our knowledge. We hope this will enable more research into MoEs by lowering the barriers of entry for researchers! [1] Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1m dollars, 2024. URL https://arxiv.org/abs/2404.07413. [2] Qwen Team. Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters”, February 2024b. URL https://qwenlm.github.io/blog/qwen-moe/. [3] Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts, 2022. URL https://arxiv.org/abs/2211.15841. [4] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing, 2022. URL https://arxiv.org/abs/2202.09368. [5] Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints, 2023. URL https://arxiv.org/abs/2212.05055.

Thanks a lot for your detailed review - we are glad you found our analysis insightful and delivering fresh perspectives! **Regarding your question about whether model parameters are quoted from the original papers:** Yes, we intended to quote parameters directly from their papers, but there is indeed a typo in Table 2 and it should be 2.6B instead of 2.9B for OpenMoE. This number comes from Appendix B, Table 11 in the OpenMoE paper [1] and corresponds to their active parameters with embedding. 2.1B is their active parameters without embedding, but since we include the embedding parameters in the active parameters of all other models, we also do so for OpenMoE. In Figure 1, we placed OpenMoE correctly corresponding to 2.6B. We have fixed the typo in Table 2 to 2.6B and revised the submission. We have also double-checked all other numbers. Thanks a lot! [1] Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models, 2024. URL https://arxiv.org/abs/2402.01739.