HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

Summary

This paper introduces a one-stage domain adaptation technique that combines diverse data from pre-training and supervised stages into a concise instruction-output pair format for efficient knowledge injection. The proposed mechanism, according to the authors, improves both training stability and domain generalization. Extensive experiments show the effectiveness of the method in four medical benchmarks (MedQA, MedMCQA, CMB, CMExam) and two general benchmarks (C-Eval, CMMLU).

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The suggested approach seems straightforward and might be used for other domains as well. - Attempted to solve a critial issue of catastrophic forgetting in language model. - The paper performed detailed experiments and ablation study to validate the effectiveness of the proposed method.

Reasons to reject

- The model, HuatuoGPT-II, specically developed only for the medical domain in Chinese. I have reservation with the claims of training stability and domain generalization. Please refer to 'Questions To Authors' for more details. - The quality of the writing is a little sloppy. For instance, the overuse of "Camel Case," which is superfluous, detracted from the quality of writing. Furthermore, it is wrong to say '... let **an** Large Language Model (LLM) generate...' and unusual in academic writing to say '... and close to the data’s original content...'. I would like the authors to carefully review these minor issues throughout the manuscript.

Questions to authors

The authors claim that the proposed protocol improves training stability and domain generalization. What potential effects might it have on medical fields other than Chinese or other domains (e.g., law, finance and so on)?

Summary

This work presents a one-stage domain adaptation pipeline that relies on instruction tuning in place of the more common two-stage pipeline comprising continual pre-training and supervised finetuning. This approach aims to offer several advantages: a form of simplification, a means of reducing catastrophic forgetting, and mechanism to adaptively adjust data source during training. To test the efficacy of this model, there are numerous experiments (and a rich appendix) to substantiate claims.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

This work demonstrates several strengths: - There are many experiments that aim to substantiate the claims (and a significant appendix) that helps piece together several parts of the story. - The notion of simplification and adding a "knob" to more easily adjust data mixing are both quite useful in the context of model adaptation and could potentially help mitigate multiple distribution shifts using a consistent method. - The use of other tools, e.g. GPT, to construct instructions introduce an additional source of information that can be scaled more easily than the traditional data generation process.

Reasons to reject

There are some considerations that limit impact, namely: - Prior work (Peng et al., https://arxiv.org/pdf/2304.03277) demonstrated the efficacy of instruction tuning with GPT-4, which would be useful to contextualize the present work. Further, it would be useful to add discussion around whether this setup might suggest that GPT-4 offers an upper found on performance for these tasks were it similarly finetuned. On the one hand, the experimental results suggest HuatuoGPT-II has a superior win rate, but it's not quite an equal comparison. - Adding clarity around the underlying distribution shifts that are present would be useful because there is both a significant shift in language and general vs medical domain present in this work. In particular, the evaluation chosen relies on resources like MedQA which are primarily clinical questions (from medical exams) whereas the training data is drawn from biomedical (as opposed to clinical) domain. - Focus the discussion of the results to emphasize the key takeaways. Currently there is a rich appendix that substantiates several claims made throughout the text but the main paper is difficult to read in absence of the appendix because key details have been relegated there, some of which could arguably be considered more important than those that appear directly in the results, such as Appendix N (Other Medical Models), which provides context for what performance means in this space.

Questions to authors

Suggestions: - Carefully consider whether there are results that appear in the appendix that would be better included in the main text. - This is stylistic, but consider removing some parentheticals where not necessary, e.g. "(necessitating dual meticulous training)". Questions: - What are future directions can be highlighted in discussion that are revealed by differences between Huatuo and Huatuo-II? - If a different LLM is used in place of GPT-4 for generating instruction tuning data, how significantly does this impact the quality of the resulting model? - How is performance affected with a very small (e.g. 1B) model or much larger model?

Summary

The authors tackle the problem of domain adaptation of large language models (LLMs) by proposing a unified, one-stage adaptation process in the place of the more common two-stage approach of continued pretraining and supervised fine tuning. Their approach transforms the domain-specific pretraining corpus into a form that more closely resembles the instruction tuning data used during fine tuning. The authors select a filtered dataset of over 5 million documents in the medical domain to demonstrate this transformation process and how this data is used to adapt a general-domain LLM to the medical domain. The resulting model and competing baseline models are graded against a recent, previously unseen Chinese National Pharmacist Licensure Examination, with the hope that the contents of this test are not present in some form in either the general or domain-specific training data used by the models. Both automatic and human-expert evaluations are performed. Results show the authors' models generally outperforming the baseline models on Chinese-language tasks, including commercial offerings such as GPT-4.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The unified one-stage approach for tuning a LLM to a specific domain is interesting, especially where the pretraining data is transformed into instruction-tuning-like data via interactions with a separate LLM (the authors note there is no strong reliance on a single model in this transformation step.)

Reasons to reject

While there has been great care to note that the licensing exam has been chosen for novelty (and was published after the data collection cut-off date), there may questions present in earlier published study guides that were then incorporated as training or tuning data. It may be worth a quick examination on scores from previous editions of the licensing exams for comparison.

Questions to authors

- Can you define what is meant by 'vertical domains' in Section 1? - A footnote for a definition of 'Philosophy of Parsimony' may be useful - "unpopular languages" may be better represented as "lower-resourced languages" in Section 1 - Section 3.1, corpus is -> corpora are

Summary

This paper introduces a one-stage domain-adaptation protocol for language models and produces a medical domain language model HuatuoGPT-II. Different from a two-stage process (pre-training, supervised fine-tuning), the paper proposes a one-stage approach to merge these two stages. The key technique here is to unify different format of data into instruction tuning pairs, leveraged by auto-labeling ability of large models (i.e., GPT-4). The paper also proposes a priority sampling strategy for training. The models are shown to perform better compared to other models on different Chinese medical benchmarks.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

- The paper proposes a simple training protocol, which can indeed help reduce the efforts needed in a traditional two-stage training. - The paper is easy to understand. - HuatuoGPT-II shows strong performance on different benchmarks - The paper also introduces a new Chinese Pharmacist Licensure Examination to avoid the test data leakage concerns for existing benchmarks.

Reasons to reject

- The connection between training on pre-training corpora and training on instruction data generated from pre-training corpora is not clear to me. But it seems to be a core inspiration of the methodology. I think providing more and deep insights will be useful. - The baseline approaches are a bit weak as they are mostly general-domain models. - Given the paper uses GPT-4 to generate data, uses GPT-4 to evaluate results, and compares the proposed model with GPT-4, it will be tricky to interpret the results, especially when comparing to GPT-4.

Questions to authors

- What types of questions are generated on the pre-training corpus? - The paper makes an assumption that training with instruction data (which is generated on pre-training corpus) has the same functioning of training on pre-training corpus. What is the intuition behind it?

Acknowledgment on rebuttal

Thanks for your efforts in preparing the rebuttal. I have read the authors' feedback and other reviews. I would prefer to maintain my initial assessment.

Acknowledgement of Rebuttal

Thank you for answering these questions. In particular, the answers to Q5 and Q6 are quite interesting and I would encourage the inclusion of these results, whether in the work itself or an appendix if space doesn't permit. For Q2, please do try to add clarity around this point in revision but it's not necessary to do so here.

Acknowledgement of rebuttal

Thank you for providing additional information and clarifications for your work, in particular regarding performance on previous years exams. At this time, I will concur with the other reviewers and maintain the current review score.