Crystal: Illuminating LLM Abilities on Language and Code

Summary

This paper investigates the integration of natural language and coding capabilities within code LLMs. The authors introduce a novel pretraining strategy that involves two training phases with a carefully adjusted mix of code and natural language data. Their proposed model, Crystal, demonstrates comparable performance in natural language processing and coding tasks to established models like Llama2 and Code Llama, but with greater data efficiency, using 1.4T tokens versus over 2T used by the other models. The effectiveness of the new pretraining approach is supported by detailed analysis of the training process and benchmarking, which show consistent improvements.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The motivation behind this paper is strong, highlighting a critical gap in open-source LLMs which typically excel either in coding or NLP tasks, but not both. This contrasts sharply with more versatile closed-source models like GPT-4. 2. The paper provides detailed insights into model performance through the inclusion of evaluation results from different training checkpoints (Figure 5) and ablation studies (Table 3), clearly illustrating the impact of various training phases on Crystal’s effectiveness across tasks. 3. Comparative analysis reveals that Crystal achieves strong performance on both coding and NL tasks, rivaling the capabilities of similarly sized models in two domain at the same time. 4. The evaluation is comprehensive, particularly in the NLP domain, where Crystal is benchmarked against strong baselines such as LLaMa-2, Code LLaMa, and StarCoder across 11 distinct tasks. 5. Crystal shows notable computational efficiency, achieving strong performance while requiring significantly fewer training tokens than comparable models. Note: While the paper mentions a commitment to releasing training code and all intermediate checkpoints, which could be a significant contribution, this is not yet verifiable in the review phase without supplementary material. Thus, it is not included as a strength here.

Reasons to reject

1. It remains unclear whether the observed performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset. The current ablation study does not address this question. Although retraining the full model is resource-intensive, conducting a smaller-scale experiment, perhaps with a 1B parameter model, could validate the effectiveness of the two-phase approach, which is central to this paper’s contributions. 2. While the paper benchmarks Crystal against notable models, it omits comparisons with key baselines like GPT-3.5 and GPT-4, which are recognized for their proficiency in both coding and NL tasks. Additionally, including comparisons with models like Mixtral-8x7B, which also excel in both domains, would enhance the robustness of the evaluation. 3. *[Minor]* The pretraining strategy proposed in this paper, which consists of two distinct phases using different mixtures of NL and code datasets, is too straightforward and lacks significant novelty. 4. *[Minor]* The evaluation methodology appears somewhat biased towards NLP, as the evaluations on the coding aspects (3 tasks) are not as comprehensive compared with on the NLP side (11 tasks). Expanding the evaluation to include additional coding benchmarks such as APPS or DS-1000 could provide a more balanced view of the model's capabilities.

Questions to authors

See weakness 1. Is it possible to see if performance improvements are due to the two-phase training method or if similar outcomes could be achieved with a single-stage training using a combined dataset?

Summary

Authors propose a pre-training strategy that transitions from a phase of learning from natural language heavy to code heavy dataset. This has been claimed to achieve the most optimal balance between performance on natural language tasks and coding tasks for a smaller training budget than comparable work (in terms of model size) like Llama2 and CodeLlama.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

This work provide a closer look at the details of the pre-training stages of language models, which has been undisclosed in the case of most openly released models in recent times. Particularly, the stages of pre-training and their impact on downstream performance through empirical studies haven't been studied in this manner to the best of my knowledge. This can benefit the community's understanding in efficiently training models (with as small pre-training costs as possible) that are performance-wise optimal in both code and natural language domains. For instance, they highlight the variation in the loss scales across the two phases as code data is more predictable leading to a lower loss-per-token. The final model they obtain after the different training stages is indeed competitive in both domains when compared to specialised models like Llama 2 and CodeLlama that have been trained on a similar scale of data.

Reasons to reject

Authors don't discuss the motivation of needing a single model to perform well at tasks from both natural language and code domain. One possible motivation could be that Code LLMs that can follow instructions are suited for a wide variety of agentic tasks as shown in Hong et al. However, one could argue if this capability can be achieved simply by instruction fine-tuning stage on top of a code pre-trained LM. While the impact of controlling the ratio of code to natural language data during pre-training has been studied, there is no discussion on the quality of the dataset used. Recent work on phi-2 (https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/) has shown performance superior (MMLU, HumanEval, MBPP) to Crystal when trained on a similar size of data ~1.4T tokens, and using a smaller model in terms of parameter count (2.7B). Comparing this work to phi-3 or Llama 3 wouldn't be fair given the timeline of events. Hong et al MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

Questions to authors

- The CG-1 architecture is not described in sufficient detail. Besides LayerNorm what architectural changes are required for training on this non-GPU hardware? - What is the training duration? (similar to how GPU hours are being reported in other studies) - In Fig 3, can the authors explain the occurrence of spikes in the training loss during stage 1? - Why is the qk dot product scaling different compared to the commonly used $\sqrt{d}$ formula? (Table 4) - What does rotary percentage mean in Table 4? - What is the source of the 100B tokens in the phase adaptation stage? What is the motivation in including this stage? - Creating a benchmark like WebMC is an interesting contribution. Can the authors provide qualitative analysis to describe what the tasks here look like? - In 5.2 authors mention the adaptation phase to be hurting NL tasks. Could the reason be that the fraction of Slim Pajama is too little in this stage? Suggestions: - Pre-trained models like X-Gen, Phi series could be added to the related work section. XGen (by Nijkamp et al https://arxiv.org/abs/2309.03450) specifically discusses the approach of having different stages each with different sampling proportions. - Besides k-shot, authors could also try CoT prompting w/ the Crystal models to push performance on the NL and code benchmarks, particularly MATH and GSM8k. - A summary of model architecture details could be a part of the main paper (currently in Table 5 and Appendix B), similarly Table 7 should also be a part of the main paper. There is quite some whitespace (e.g. Fig 5 (could use shared legends), Fig 6, Section 6) that could be removed to achieve this compression. Minor typos/linguistic issues: - Abstract: Futhermore, there lack of --> there is a lack of? - Section 3 Phase 2: we intent to --> we intend to? - Section 5.2 Para 2: adaptation gains are even out

Summary

The authors introduced a pretraining strategy that focuses on training LLMs with both text and code data. The resulting family of models, CRYSTAL, is stronger in both natural language and coding performance, compared to strong baselines like LLama2 and Code Llama. The authors are also committed to releasing open-sourced model checkpoints, increasing the replicability and transparency of LLM training.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- Despite pretrained on a smaller set of tokens (1.4 trillion), CRYSTAL seems to possess strong/comparable performance against other models pretrained on a larger dataset (2 trillion tokens). - The evaluation is done comprehensively on many language tasks (reasoning, QA, general knowledge, etc.) and coding tasks (HumanEval, MBPP, WebMC). Interesting ablation results (e.g. data efficiency, training curves) are analyzed quite well, bringing useful insights to LLM training. - The authors’ commitment to open-source the models is commendable as it will help to push innovation and collaboration within the research community.

Reasons to reject

- The use of a mixture of datasets of code and text over multiple stages of training is not so new. For instance, CodeT5+ (https://arxiv.org/abs/2305.07922) was also trained on both code and text data over two stages of pretraining. While the model was not evaluated on conventional NLP tasks, it performed quite well over multimodal test setups such as code-to-text and text-to-code benchmarks. Can the authors clarify what is the main novelty compared to CodeT5+? Have you tried to benchmark CRYSTAL in more diverse code-text tasks beyond HumanEval and MBPP? - Pretraining models from scratch is expensive. I wonder if the current pretraining strategy can be applied on top of any pre-trained models. If subsequently similar insights could be observed, I think the current method would be a lot more impactful.

Questions to authors

- Typo (Introduction): “pertaining” - A useful ablation the authors can try is to use the data in phase 1 or phase 2 but train for the same number of training steps as phase 1 + phase 2 training combined. This would isolate the effect of training steps and demonstrate better how the data mixing contributes to the performance.

Response to rebuttal

I thank the authors for the clarifications provided, and am revising my score accordingly.