Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

Summary

This paper proposed a new method for LLM alignment during pre-training. The proposed method is call "native alignment". This method include three steps: pretrain date duplication, alignment rewriting, and model training. They trained small size alignment expert model for alignment rewriting and use the model to rewrite large-scale pre-training data. The rewriting process suppose to solve format issue, value/fairness issue, unsafe content in pre-training data. They experimented with Arabic data and LLMs. Their experiments shows that the proposed method can help LLMs be more safe and helpful.

Strengths

1. The paper proposed a new idea to align LLMs during pre-training. It seems an interesting topic. 2. The paper writing is clear and well-organized.

Weaknesses

1. Lack of comparison to existing post-alignment methods. The proposed method is a "native alignment" during pre-training. I wonder if this method can outperform the post-alignment methods. While the author acknowledged this limitation, I still feel it is important for strengthening their claim. 2. Need more analyses to better understand their method's potential trade-off. For example, I wonder if rewriting pre-training data undermines the LLM's capacity to understand and learn Arabic dialects. The rewriting process may convert Arabic dialects into MSA. I also wonder if the rewriting data inherited hallucinations from LLM and deteriorated the trained model. 3. The paper needs more clarification on experiment details. For example, they exploit Arabic data to investigate their method, however, the evaluation dataset, BeaverTails dataset, is an English dataset. I wonder how they evaluate and if they translate the samples.

Questions

Please see the questions in weakness. * I wonder whether you continued training the LLaMA 3 model or newly initialized LLaMA-like model and trained it from scratch.

Rating

6

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

I think that the paper needs more diverse analyses to understand the potential trade-off of their method.

Summary

The paper introduce a method called "native alignment", which is a set of procedures to create data and train an LLM to rewrite raw text into "useful" texts for pretraining. They apply this technique specifically for Arabic LLMs and conduct experiments to show that this pre-processing of pre-training data helps produce better Arabic LLM down the line. As bonus, they release open-source Arabic LLMs for the communities

Strengths

* The paper ideas are presented clearly and easy-to-understand

Weaknesses

* As a proclaimed novelty, the paper draws itself between pre-alignment and post-alignment, indicating that previous work only focus on post-alignment but not pre-alignment. However, I afraid the paper misunderstands the concept of post-alignment (RLHF) and fails make an accurate comparison. Post alignment (RLHF) is finetuning technique to train the models to reward good-vs-bad response according to human values, and train the policy models to lean on the good behavior and stay-away from the bad behaviors gradually, often with the existence of a reference model (DPO and RLHF). Meanwhile, the "native alignment" presented in the paper is a data-cleaning procedure, and it does not having any resemblance or contrast with "post-alignment". Furthermore, using LLMs or training LLMs to rewrite raw text to produce cleaner data is not new or novel, there are many techniques out there that do so, and there are abundant open-source data on huggingface which were produced in similar ways. This confusion between data cleaning and alignment makes the paper less credible and the lack of novelty it the methodology itself, as a data cleaning method, is also troublesome. Obviously as a result, the paper did not provide any necessary and required experimental comparisons with other data cleaning methods. * Though I do appreciate the paper's effort for Arabic community, the scope of only Arabic LLM is small and generally inconclusive, that such method is not shown to generalize to other languages, domains. Perhaps, thus, the work is really not suitable for NeurIPS but more suitable for CL-type venues * It is unclear from the writing whether the authors pretrained Llama-3 with Arabic from scratch or further finetune from Llama-3 checkpoint. In either case, there should be explanation and further ablation studies.

Questions

Did the authors pretrain from scratch (with Llama-3 architecture) or from Llama-3 checkpoint

Rating

3

Confidence

5

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors discussed limitations

Summary

This paper proposes a data augmentation pipeline which modifies the pre training data for large language models in key aspects such as formatting, values, content moderation and knowledge preservation. The resulting pipeline, termed native alignment, is applied Arabic LLMs due to the relatively small pretraining corpus available and the difference between Arabic and western culture. Experiments are conducted to test the performance on a few metrics including trustworthiness, knowledge, and Arabic localisation.

Strengths

This is a well written paper targeting the important topic of llm alignment. It also addresses the relatively under explored sub question of how to improve alignment at pretraining. The resulting pipeline presents a reasonable idea, and the evaluations are clear and I find them comprehensive too. The author(s) should also be commended for their transparency regarding the limitations of the paper.

Weaknesses

Although this might have become the norm of recent LLM papers, I still think it is important to include a discussion of the metrics used to measure things like 'trustworthiness' and 'knowledge', as these are qualitative metrics, whereas in the paper, it seems like the authors just quoted some existing evaluation pipeline.

Questions

Step 3 of the pipeline talks about training language models to act in place of the human experts. I may have missed this but I think the authors should explicate how exactly this is done in the experiment section - are the authors using already pre trained LLMs to finetune as experts? How would we know that these are aligned themselves? If we cannot trust the LLM experts and must resort to human experts, then it's unclear to me how this method should scale up. In the experiment section the authors show that LLMs pertained on both the original pretraining data as well as native aligned data work better - how does one interpret this result? Since, if the original pretraining data contains harmful or value-misaligned data points, then it seems reasonable that the LLM does not learn from these at all.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

As the authors are already open about, comparisons with other post alignment methods are not included. The authors attribute this to an absence of existing alignment evaluation benchmark, but I don't fully understand this - what is stopping the authors from using the same alignment benchmarks as ones they have already used to compare with other pretrained models?

Summary

This paper focuses on alignment of LLMs to human preferences and suggests to shift the alignment step from instruction-tuning (post-alignment) to the earlier stage of continued pre-training (native alignment). For that end it proposes an approach to creating aligned pre-training data, consisting of three steps: (1) seed data cleanup and rewriting with humans'/LLM help, (2) training a supervised cleanup model on that seed set and (3) processing the final pre-training dataset with that cleanup model. Presented experiments show that alignment data results in higher final quality compared to unprocessed pre-training data and that the performance gain does not reach a plateau at 12B tokens, suggesting that the amount of alignment data should be limited by the budget allocated to train an LLM. Experiments are performed on Llama-3-8B and Llama-3-70B and the Arabic language.

Strengths

- A high-impact and efficient approach to pre-aligned model training is introduced - Two pre-aligned LLMs for Arabic are released openly based on the experiments in this paper - Related work is excellent, the paper is written very clearly and is easy to comprehend

Weaknesses

1. No direct comparison between native alignment and post-alignment is reported 2. Minor text discrepancies are present: - rows 16-18: partial sentence "while.." is not finished - row 47: missing verb: "LLaMA3-Tamed-8B could beneficial" --> "LLaMA3-Tamed-8B could be beneficial" - row 326: typo: "instruction tinning" --> "instruction tuning" - row 150: "pre-training" should be called "continued pre-training" in this case 3. The created seed data and cleanup models are not released

Questions

Q1: In the description you juxtapose native alignment and post-alignment, yet there are no experiments comparing their effect directly. What is the basis for claiming that native alignment yields better results in terms of helpfulness, harmlessness or other metrics? Q2: Hypothetical question: should we as community not aspire to create top-performing models beating GPT4, not create the best models _under_ it, led by it?, more specifically, in your setup of experiments and model training, is the final result bound by GPT4's performance, or can it surpass it?, why hasn't it, according to Table 2? Q3: How much in your opinion does the choice of seed data and synthetically cleaned alignment data affect the results?, would you consider any approaches to select these sets non-randomly, either directly or via some version of active learning? Q4: Why not release your seed data, curated by GPT4?, perhaps also the cleanup models, or even the 12B set of generated alignment data?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

Ok