Zephyr: Direct Distillation of LM Alignment

Summary

This work produces a 7B large language model Zephyr, using distilled supervised fine-tuning and distilled direct preference optimization for alignment via AI feedback. Specifically, the authors first conduct dSFT on filtered UltraChat and dDPO on UltraFeedback (1-3 epochs) for LM alignment, based on the powerful Mistral 7B. Extensive experiments on different datasets and settings have been conducted, compared with various dPPO, RLHF, dSFT LLMs of different sizes. In conclusion, Zephyr 7B achieves impressive performance on most tasks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1) This work explores the distilled direct preference optimization technique for LM alignment based on AI feedback, which could facilitate future research. 2) The performance of the proposed 7B model is impressive. 3) This idea is sound and easy to follow. 4) Extensive experiments on different datasets and settings make the results more solid.

Reasons to reject

1) The effectiveness of dDPO could be further verified on different LLM sizes and base LLM models. 2) More details on the specific data preprocessing strategies on UltraChat and UltraFeedback should be discussed (better with more quantitative experimental analyses). 3) More discussions on the side effects of considering safety in dDPO on other abilities (e.g., chat, reasoning in Table 3) should be given.

Questions to authors

1) More analyses on the dataset preprocessing of UltraChat/ UltraFeedback are encouraged to be released. For example, the authors randomly select one of the remaining three low-scoring responses of other models as the “rejected” response. Although the authors claimed that the randomness could improve the diversity and hardness of DPO. Is there any quantitative experimental analysis on this selection? Readers may want to know to which degree the diversity of low-scoring samples affects the performance of dDPO. 2) This work only conducts dDPO based on Mistral 7B. It is interesting to verify the effectiveness of different LLMs. 3) The authors have given some related works to preference alignment released after this work. The main differences between Zephyr and these works could be clarified. 4) In Table 3, will adding more consideration on Safety in dDPO lower the performance of other aspects?

Summary

This paper presents Zephyr which achieves state-of-the-art performance on chat benchmarks for 7B parameter models and outperforms the LLAMA2-CHAT-70B model. The model is built upon Mistral-7B with SFT on UltraChat and DPO on UltraFeedback.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

- The Zephyr models are quite good. I believe this alignment recipe will benefit the open-source community. - The experiments are solid. The authors provide sufficient experiment details and ablation studies, making the reproduction much easier.

Reasons to reject

- This paper is more like a technical report than a research paper. It does build a powerful open-source model, but there are limited research contributions. The core components, Mistral model, UltraChat, and UltraFeedback datasets, are proposed by others. - The proposed terms dSFT and dDPO are unnecessary. They are still standard SFT and DPO, with merely different datasets. Overall, I appreciate this paper for its empirical openness, including open-sourcing all code, data, model weights, and training recipes. However, the technical novelty is rather slim.

Summary

This paper technically introduces how to produce a series of large language models within 7b parameters, based on open-source models, open-source distilled SFT and preference datasets. Comprehensive evaluations on different popular benchmarks are available.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1. The presentation is clear and easy to follow. 2. Detailed experiments and analysis. I believe the models and the paper are technically good contributions to the community.

Reasons to reject

It’s fine that Zephyr uses DPO for alignment and doesn’t have a PPO version. However, the authors denote that they compare “PPO approaches” in the experiments. I feel it’s a bit overclaimed since the dPPO model Xwin-LMv0.1 is based on a different model llama2.

Questions to authors

Do you have plans to try larger models?

Summary

This paper proposes a training pipeline to achieve the performance of 40B+ scale large language models (LLMs) using a relatively small 7B model. Through three stages—dSFT, AIF, and dDPO—the authors transform Mistral-7B into the powerful Zephyr model, which outperforms various other 7B models. The effectiveness of the proposed training path is demonstrated through extensive performance comparisons with various LLMs.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- Proposes effective training pipeline to develop a small yet powerful model. - Provides detailed explanations of the experimental setup. - Conducts thorough performance comparisons with various models and meticulous ablation studies.

Reasons to reject

The paper lacks information on the cost involved in creating Zephyr. Details on the costs of using LLMs to create datasets in dSFT, generating answers with multiple models in AIF, and ranking responses using GPT-4 would help in understanding the practical feasibility of the proposed method.

Questions to authors

The paper can be seen as a form of imitation learning. While the significant improvement in chat performance is understandable, the substantial increase in academic benchmark performance shown in Table 2 is intriguing. What insights can the authors provide to explain this phenomenon?

Thanks for the reply

thanks for the rebuttal. Given the already high score, I’d like to keep it.

After rebuttal

Thanks for the authors' rebuttal, which has answered most of my questions. Considering the relatively high score I've given, I will maintain my voting of Accept.

Re

Thanks for the reply. I agree that bringing pieces together is important, and the Zephyr recipe has been a standard approach for open-source alignment. However, the criteria in the open-source community and academia are different. Similar popular open-source models like LLaMA, Mistral, and Phi have never been published in conferences or journals. Therefore, since other reviewers already gave high scores, I insist my score to provide a different view.