PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications
Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct doctor downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. Our model and dataset will be open-source for community development.
Paper
References (58)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The authors of this paper built PedCorpus, a Chinese pediatric dataset, and PedistrcsGPT, the first Chinese pediatric LLM assistant. Their model was built via continuous pertaining, full-parameter supervised fine-tuning (SFT), direct following preference optimization, and parameter-efficient secondary SFT. Performance evaluation results show that their model outperforms other models.
Strengths
Originality: This is the first time that Chinese LLM has been applied to pediatrics. The author also proposed DFPO, which improves the performance of preference optimization. Quality and Clarity: Generally, the paper clearly presents the datasets, the methodology, and the performance evaluation, including the experiment settings and results. Significance: The model could complement the current shortage of healthcare resources in pediatrics.
Weaknesses
1. The techniques in this paper are not that novel, as it applies LLM to another field with main techniques either not original or adapted from others (e.g., DFPO). 2. The paper focuses on the application level. I believe the model could be important if it is deployed. However, there is no evaluation from real users (patients or patients' parents). Doctor's evaluations could be different from the patients' evaluations, especially since this model will be used for children. This weakens this paper's contributions. I think the authors should at least discuss their plan of deployment or intervention as future work.
Questions
1. Do you think DFPO performs better only on your specific task or can generally perform better than RHLF and vanilla DPO? If it can generally perform better, then this can be a great contribution. 2. How will you deploy your model in practice to complement the current shortage of pediatric healthcare resources?
Rating
7
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
The authors list two limitations, which make sense. I wonder whether your model can be 100% accurate. If it generates errors or hallucinations, it should be important to find them and avoid showing them to patients and their parents. As your model will be used for medicine and children, this should be more important than the LLM applied to other fields. This again goes back to my question before: how will you deploy your model in practice?
Summary
- This paper introduces PediatricsGPT, a Chinese AI assistant for paediatrics - They created a large dataset (PedCorpus) with 300k+ medical instructions - The training process is pretty involved - includes pre-training, fine-tuning, and preference alignment - They came up with some new techniques, like hybrid instruction pre-training - Evaluation was thorough - used metrics, GPT-4, and even had real doctors test it - Results show it outperforms other Chinese medical AI models in various health tasks, as well as ChatGPT - The authors discuss its potential to support doctors and improve pediatric care
Strengths
- It's tackling a real problem - the shortage of pediatric care in China - The dataset they built (PedCorpus) is very comprehensive and high-quality - Their evaluation is thorough, using multiple methods including real doctors - The model outperforms existing Chinese medical AIs, which is impressive
Weaknesses
Overall the paper would benefit from more information on dataset construction, for each specific task and in general. In many cases, it is not clear what is meant by manual sampling, automatic extraction, dataset collection and similar. It would be great to see the standard deviation for the reported results. It would be great to see a comparison with human experts, but this is too much to ask for now, so future work.
Questions
Line 12: Immediately, the... Please rephrase, a bit hard to understand Line 29: Please either reference your own results, or cite a resource that supports your claim. Line 101: Please add information on how was this done, how were the books selected and how was the knowledge automatically extracted. Line 109: Given that real patient conversations were used, it would be beneficial to know from where exactly was this data obtained, and were the appropriate ethical guideliens followed. Line 107: Manually sampled - does this mean a human went through everything and selected the appropriate data. Line 164-168 - PedCorpus-DFPO is selectively sampled from vanilla PedCorpus - What does it mean "selectively sampled", how was this done? Were there any biases? What parts of PedCorpus were sampled and in which proportion? - Humanistic stylistic rephrasing - what exactly is this? Who performed the rephrasing (humans or AI, or a combination)? What guidelines were used? How was the quality and consistency of the rephrasing ensured? - Low-capability medical assistant - please add a bit more info on why was the HuaTuo model chosen, and why is it low capability (how was this verified). Line 208: How were the 208 difficult tasks sampled, and how was difficulty determined? The paper would benefit (especially the first couple of pages) from a review of the grammar and sentence clarity. Note: most of the datasets are in Chinese, maybe some information was obvious, but because of my lack of understanding of the language I was unable to determine that.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
A very limited validation on real-world use cases, this is more of an initial research paper that can be built upon and further tested in the real world. Would be great to address this in the main part of the paper and not an appendix.
Dear Reviewer eEmB: We would like to thank the reviewer for taking the time to review our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us. Best regards, Authors
Summary
This paper tried to build an AI-powered pediatric consultation system. Their motivation is current Chinese conversation LLMs for healthcare underperform in pediatric applications due to insufficient instruction data and bad training procedures. To tackle this problem, the authors proposed PedCorpus, a high-quality dataset with over 300K multi-task instructions, and PediatricsGPT, the first pediatric LLM assistant in Chinese. The PediatricsGPT baseline models, enhanced with a hybrid instruction pre-training mechanism, supervised fine-tuning, preference optimization, and a strategy to balance general and pediatric expertise, consistently outperforms previous Chinese conversation LLMs.
Strengths
- The authors are inspired by the new progress in LLMs and then build new datasets for Pediatrics application and new domain-specific LLM model weights for this task. - The paper's structure is clear and the authors propose a rigorous evaluation pipeline.
Weaknesses
- "Direct Following Preference Optimization (DFPO) in human intention alignment is devised to enhance response robustness and align human preferences." - where is the evidence for this? It would be better to visualize some cases other than using ablation study as we can not see a huge performance difference. - "ternary instances in the knowledge graphs" How to build the ternary instance for your task. - "complementary resources" What is a complementary resource? - Not sure how to protect patient privacy in your Real Doctor-patient Conversations subset. - As a pediatrician, I don't think the current model has "competence in different pediatric and general healthcare service scenarios". I review your data in the supplementary, it seems most of data do not contain real pediatrics AI challenge. To prove the generalization of the proposed model or prove the value of the dataset in Pediatrics, you should consider building datasets with more specialized pediatric diseases such as autism, cerebral palsy... - It is also important to test the performance of proposed model in other Out-of-Domain dataset, such as other medical QA dataset for adult. - It is confusing in your dataset building process. Why choose “expert pediatrician” GPT-4 but not real pediatrician to help you construct the dataset? - "doctor-like and patient-friendly" - how to define doctor-like and patient-friendly. Do you mean 'professional' here? - The scale of user study is limited. You state "We invite three doctors (each paid $300) to determine the winner of pairwise models by the majority voting rule." However, it would be better to conduct a wide user study from pediatricians as three people can not represent the human preference of the whole group. - Institutional Review Board (IRB) Approval is needed as this paper has human user study and the released data includes real doctor-patient conversations. - Some minor issues: "Chinese medicine" can be replaced by healthcare service in Chinese as its clinical meaning can be found here: https://www.hopkinsmedicine.org/health/wellness-and-prevention/chinese-medicine
Questions
You can check the Weaknesses part and try to answer the question proposed in each item. As a physician, I agree this paper has enough novelty and can serve as a useful foundation model in the future. Thus I tend to give Borderline to this paper before reading the rebuttal. However, there are still plenty of issues that need the author to solve them. During the rebuttal, the author needs to solve the lacking issue of IRB Approval first. If IRB approval is not attached, the dataset and model may contain ethical issues and can not be released. I am also curious how ChatGPT achieves good medical and pediatric QA performance (only slightly lower than the proposed PediatricsGPT model) with in-context learning.
Rating
4
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
The main limitation is lacking IRB Approval. And most of the data do not contain real pediatrics AI challenges. This dataset can only be considered as a supplement of current released medical QA data for adults in China such as DISC-MedLLM.
Dear Reviewer eZYU: We would like to thank the reviewer for taking the time to review our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us. Best regards, Authors
Dear Reviewer eZYU: As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback! Best regards, Authors
Dear Reviewer eZYU: As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback! Best regards, Authors
Thanks for the comment and we will make further clarifications about humanistic responses.
**Q3**：Whether the conversational responses in the dataset support the assertion that the model produces humanistic responses. **A3**: We would like to clarify that Appendix A.1 and Figures 7\&8 show how valid instruction pairs can be extracted from streamed text from specialized pediatric data. At this stage, we expect the role-play driven API to build instructions with as much guaranteed knowledge completeness and accuracy as possible, rather than contextually adapted answers. Adapted answers will largely suffer from the API's own hallucination problem, leading to potentially noisy and non-factual information. After this stage, we feed the built initial instructions back to the pediatricians to help us with humanistic stylistic rephrasing, referring to making the responses as caring and well-presented as the pediatricians’. We promise to supplement that the humanistic style comes from the physicians' rewrites after rather than extracting the preliminary instructions in the revision.
Thanks for the comment and we will make further clarifications about similar solutions.
Dear Reviewer eZYU: We deeply appreciate the reviewer's comments. Based on your response and original comments, our rebuttal appears to have addressed the vast majority of your concerns. We would like to clarify the remaining concerns as follows. **Q1**: The reviewer recognizes and appreciates the thorough evaluation and novel training pipeline. However, the reviewer affirms that there are similar solutions in the generalized domain. **A1**: We are concerns that we do not fully understand your insights into similar solutions. It would be helpful if the reviewer could present specific examples for reference that would help explore the interpretation and provide evidence for this statement. Here, we go through the content to help the reviewer address the concern. We start by listing some of the other reviewers' comments to help you come to a consensus with them, as you hope to do in your response. As described by the **reviewer oou4**, our innovative training process, which includes continuous pre-training, full-parameter supervised fine-tuning, preference optimization, and mixture of universal-specific experts-based supervised fine-tuning, effectively adapts the model to the pediatric domain. The **reviewer jQHJ** explicitly noted the multi-stage training process of this work helped enhance the model's ability to produce good responses and adapt to the complexity of pediatric consultations. In addition, The **reviewers eEmB** and **MG71** pointed out multiple domain-specific innovations in our systematic training pipeline in the strengths, including the new preference optimization strategy DFPO and the hybrid instruction pre-training mechanism. Of course, these new contributions we have also accomplished explicit clarifications in the manuscript. Specifically, * As stated in lines 131-138, we propose a hybrid instruction pre-training mechanism in Continuous Pre-Training (CPT) to bridge the capability weakening due to corpus format discrepancies between the internal and injected pediatric knowledge of foundation models, facilitating knowledge accumulation and extension. Our mechanism better facilitates the adaptation of model capabilities in pediatric healthcare. * As stated in lines 169-176, we present a stable method for domain-specific LLMs called Direct Following Preference Optimization (DFPO). DFPO utilizes variable changes to formulate the preference loss as a policy function that efficiently optimizes the policy with a simple binary cross-entropy objective. Meanwhile, our method directly regularizes model behavior boundaries in an instruction-following paradigm on medical demonstrations of preferred responses, facilitating robustness and smoothing of the preference learning. * As stated in lines 186-199, we devise a mixture of universal-specific experts strategy to tackle the competency conflict between medical generalist and pediatric expertise in secondary SFT, which strengthens the model's adaptability to distinct downstream tasks. Specific experts master different pediatric tasks by soft routing gating control with noise. Also, a universal expert is consistently activated to prevent general knowledge forgetting and mitigate competency conflict. Our strategy enables our models to perform more comprehensively and competitively in pediatric medical applications. We believe that the above contributions can significantly differentiate and strengthen the necessity of our work compared to techniques in the generalized domain.
Thanks for the comment and we will make further clarifications about the primary contribution.
**Q2**: Given that your model and data are positioned as the first Chinese pediatric LLM assistant, the primary contribution of this paper should be the quality of the dataset. **A2**: As stated in lines 33-43, this paper addresses the shortcomings at the dataset and framework level of the LLM construction in the Chinese healthcare and pediatric domains. We propose a high-quality dataset PedCorpus through a series of new instruction-building mechanisms and a systematic training pipeline through new strategies to improve the different challenges in multi-stage training. Our technical novelty is distributed among these mechanisms and strategies. For the dataset dimension, existing instruction data typically involve vanilla rephrasing of the general medical corpus or aggregation of doctor-like dialogues, which loses the specialization and focus in pediatric applications. More importantly, the current straightforward different round instruction construction paradigms fail to accommodate multi-task healthcare services in real-world scenarios, limiting the model generalization. In comparison, the novelties of our PedCorpus derive from three characteristics. * Task Diversity: besides containing generalist healthcare data, three application-oriented tasks are considered, including medical question-answer, evidence-based diagnosis, and treatment recommendation. * Source Richness: distinct pediatric textbooks, guidelines, and knowledge graph resources provide solid assurance of medical knowledge's accuracy. * Instruction Extensibility: vanilla instructions can be readily extended to seed instructions for generating specialized corpora to serve different training phases. For the technique dimension, prior methods relied on SFT to compensate for medical instruction following capabilities, ignoring the discrepancies between inherent and externally absorbed knowledge within the models. This single pattern causes secondary LLMs to lapse into excessive role-playing rather than understanding. Despite a few attempts in the reinforcement learning from human feedback phases, their performance is restricted by actor-critic instability and online sampling bias. In addition to the new techniques in the training process mentioned in **A1**, we propose three effective methods in ensuring the quality of the dataset. Specifically, * As stated in lines 106-107, we present a role-playing-driven instruction building mechanism to convert the consolidated textual data into well-designed pediatric instructions. The proposed approach endows the advanced language model with the professional role of an expert pediatrician to generate accurate instruction knowledge for the target model training. * As stated in lines 113-155, We introduce a in-context learning paradigm to regularize and refine vanilla conversations through advanced language models with the self-instruct pattern。 * As stated in lines 123-124, we design a progressive instruction reconstruction mechanism to distill the sampled instructions to ensure informative pediatric model responses. Unlike the traditional self-instruct pattern of using APIs, our mechanism guides APIs to take the perspective of the experienced pediatrician to complete progressive refinement tasks in the given instruction and answer scenarios. Based on the above analysis and explanations, our main contributions are distributed among dataset construction and framework structure, and the different mechanisms and strategies in these two aspects help our work perform better on task-specific pediatric domains and on general purpose medical applications.
Thanks for the comment and we will make further clarifications about the model and dataset.
**Q4**: The model and dataset rely on previous work such as DISC-MedLLM [1]. **A4**: We clarify the differences between our approach and previous work in terms of both the model and the dataset. **Model Level**: Current Chinese medical LLMs are constructed based on open-source foundation models for secondary development. In this case, we design a systematic training pipeline that incorporates the new direct following preference optimization (DFPO) and the mixture of universal-specific experts (MUE) structures, both of which are absent from previous Chinese healthcare models. Our DFPO facilitates the control of model behavioral boundaries, mitigating potentially harmful and unfaithful outputs. Our MUE structure effectively resolves conflicts between different pediatric tasks and the general medical knowledge, facilitating better knowledge learning and instruction following. In comparison, **DISC-MedLLM [1] does not have extra innovations in model structure and simply uses Baichuan-13B-Base to execute SFT procedures to develop medical competencies**. **Dataset level**: We elaborate on our PedCorpus and DISC-Med-SFT from DISC-MedLLM to emphasize that the two are completely different. As described in Section 3.1\&Table 1 of the manuscript, and the **global response**, the proposed PedCorpus consists of three parts, including pediatric data, real doctor-patient conversations, and distilled medical datasets. * The pediatric data comes from instruction pairs that we extracted from scratch from specialized textbooks, guidelines, and knowledge graphs, which are key to building models about task-specific pediatric applications.Among these, the knowledge graphs were constructed from scratch by ourselves from the pediatric domain. In contrast, DISC-Med-SFT uses a publicly available knowledge graph, CMeKG, which focuses on general medical knowledge and differs from our data sources and modeling philosophy. * Our doctor-patient conversation data come from two components: voice transcriptions collected at our partnering healthcare institution and the publicly accessible dataset cMedQA2. For the former, we have clarified the relevant ethical issues in the global response and given privacy protection and implementation schemes that match the NeurIPS guidelines. For the latter, we have regularized the vanilla doctor response using a context-learning strategy, which leads to more organized and fluent responses compared to the vanilla instructions. In contrast, DISC-Med-SFT uses the publicly available MedDialog and cMedQA2, which adapts the entire dialogue through prompt engineering. We would like to emphasize that incorporating cMedQA2 in the construction of medical LLMs is a public practice rather than a particular work-specific design. However, the different paradigms for processing vanilla instructions by the different works resulted in completely different final SFT data. These observations and measures imply that doctor-patient conversation data in PedCorpus and DISC-Med-SFT are significantly different. * For other datasets, we found numerous unclear and incomplete representations in the instruction instances from public benchmarks due to the absence of careful calibration, potentially triggering hallucinated outputs. To this end, we sampled a small set of knowledge-intensive instructions following the principle of quality over quantity. We introduced a progressive instruction reconstruction rule to distill the input and output parts of instruction pairs separately, resulting in completely different data compared to vanilla instructions. In contrast, DISC-Med-SFT primarily uses publicly available data from MedMCQA, moss-sft-003, and Alpaca-GPT4. Depending on the citation format we provide, ACs and reviewers can readily access DISC-MedLLM to explore our statements and clarifications. [1] Bao, Z., Chen, W., Xiao, S., Ren, K., Wu, J., Zhong, C., ... & Wei, Z. (2023). Disc-medllm: Bridging general large language models and real-world medical consultation. arXiv preprint arXiv:2308.14346.
About DISC-MedLLM
Dear Author of Paper 385, I believe there may be some misunderstanding regarding the reason I referenced DISC-MedLLM in my earlier comments. My intention was to highlight that while there are several datasets for medical LLMs with similar formats to yours, the key contribution, as you have stated, should be the quality of the dataset, particularly in the specialized field of Pediatrics. Upon reviewing the data examples provided in both your paper’s appendix and the rebuttal document, I’m concerned that the dataset may not align with what pediatric practitioners would find most valuable for their domain. This is my areas of expertise. DISC-MedLLM encounters similar issues, which is why there’s resistance from healthcare professionals toward over-generalized declarations made by these models. For instance, the data point in your appendix: "My child fell while playing a few days ago and complained of severe pain in the lower back. After visiting the hospital, the diagnosis was confirmed as lumbar disc herniation." The model's response: "Lumbar disc herniation is rare in children, but treatment still requires caution. For pediatric lumbar disc herniation, pharmacological treatment mainly includes: 1. Nonsteroidal anti-inflammatory drugs (NSAIDs), such as ibuprofen or acetaminophen, can relieve pain and reduce inflammation; ... " Similarly, in the rebuttal pdf: "I'm glad to help! Pediatric cerebral palsy is a condition caused by a non-progressive brain injury that affects a child's movement and posture. The goal of treatment is to maximize the child's physical functioning and quality of life. Treatment is usually a multidisciplinary approach that includes the following: 1. Physical therapy: Specific exercises and activities are used to build muscle strength and flexibility and to help the child learn correct posture. 2. Medication: Childhood doses of medications are used to help control symptoms, including anti-spasticity medications such as baclofen, dantrolene, and botulinum toxin injections...... 3. Surgery: Selective posterior spinal rhizotomy or muscle tendon lengthening surgery is required for some children with severe muscle spasticity...... Each child with cerebral palsy is different, and the treatment plan should be individualized and tailored to the child's specific needs and abilities." These responses resemble textbook excerpts or clinical notes, which may not be practical for real-world pediatric practice. This could lead to a significant disconnect between the machine learning and pediatrics communities, something we should aim to avoid. To address this gap, I strongly recommend consulting with pediatric specialists (your clinical collaborators) to review your dataset. Their insights would be invaluable in ensuring that the data reflects the requirements of real-world clinical practice, rather than just verifying the accuracy of the model's reply. Best, Reviewer eZYU
Thanks for the comment and we will make further clarifications about the practical needs.
**Q5**: Different pediatric departments were consulted to meet the practical needs of the researcher. **A5**: We agree with the reviewer's suggestion. Considering the multifaceted tasks and diagnostic needs in real medical applications, we believe that there is no method currently available that ensures complete generalization to address the challenges in all applications. However, we have the ability to do a good capacity building of models for practical needs on several specific tasks. With the suggestions and needs of collaborating healthcare institutions, our work focuses on three applications in pediatric healthcare, including knowledge question-answer, evidence-based diagnosis, and treatment recommendation. Correspondingly, we constructed novel MUE-based SFT processes through task-specific data for learning and mastering relevant knowledge. Also, we incorporates 131 pediatric diseases from 11 broad categories from textbooks to provide the pediatric knowledge needed for these tasks. From Table 2 and Figures 2\&3, the experiments on the three task-specific testing benchmarks clearly demonstrate the necessity of the model in solving these pediatric tasks. More importantly, the doctor evaluations in Figure 3 also reflect the users' favor and approval of the model responses. Further, we provide more participants in Figure 1 of the **global response** to demonstrate the model effectiveness on more user studies. In addition, we also test the model's generalizability on publicly available healthcare benchmarks webMedQA and CMD with multiple departments, and provide corresponding insights and analyses. As an exploratory research study, we believe that the current contributions and work are noteworthy and valuable. We are committed to collaborating with more pediatric departments and meeting further needs and tasks in our future plan.
Dear Reviewer eZYU: We appreciate the time and effort you have put into reviewing our work. We value the hard-earned time for discussion to address your concerns and make necessary clarifications. We are ready to discuss at any time. Best regards, Authors
And about the Ethic reviews
I think it is also important to discuss related information about the IRB, dataset released timeline with the ethic reviewers, because the dataset is the biggest contribution of your paper. Can you provide them related documents? And we also need the ACs to encourage the ethic reviewers paticipating the discussion. Best, Reviewer eZYU
Summary
This paper introduces a Chinese pediatric LLM assistant, PediatricsGPT. It follows a standard pretraining and SFT pipeline to incorporate the general medical knowledge schema into the models. Specifically, they optimize the response to enhance the generation of pediatrician-like humanistic responses.
Strengths
1. The establishment of a large-scale, high-quality multi-task medical instruction dataset, PedCorpus. 2. The development of a Chinese pediatric LLM assistant with pediatric expertise and medical generalist, PedatricsGPT. 3. Open source resources for the community. 4. Extensive experiments on both domain-specific capability and general medical capability. 5. Human (expert) study for a comprehensive evaluation
Weaknesses
1. The authors claim that they crafted 100 high-quality examples to guide the advanced language model, using in-context learning to regularize vanilla conversations in the self-instruct pattern, ensuring doctor-like and patient-friendly model responses. I would like to see a rigorous evaluation of this. 2. According to Table 3, MUE, Universal Expert and RLHF seems to result in a limited performance gain.
Questions
1. When designing such an LLM for pediatrics, is there any specific design/technique regarding such a target group, compared with designing a general medical LLM? I understand the preference optimization for the generation of pediatrician-like humanistic responses is one of such designs; however, is there any other specific design in the training/SFT/knowledge integration phase? 2. In Figures 2&3, how consistent is it for each question in GPT and doctor evaluation?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
4
Limitations
See weakness
Summary
The paper presents PediatricsGPT, a specialized large language model (LLM) designed to assist in pediatric medical consultations in China, where there is a significant shortage of healthcare resources. PediatricsGPT is built upon PedCorpus, a high-quality dataset containing over 300,000 instructions from pediatric textbooks, guidelines, and knowledge graphs. The model undergoes a systematic training process that includes continuous pre-training, full-parameter supervised fine-tuning, human preference alignment, and parameter-efficient secondary supervised fine-tuning. This comprehensive approach aims to address the inadequacies of existing Chinese medical LLMs in pediatric applications.
Strengths
1) The development of PedCorpus, a robust dataset with over 300,000 multi-task instructions from credible pediatric sources, ensures that PediatricsGPT has access to diverse and accurate medical knowledge. 2) The multi-phase training process, which includes continuous pre-training, full-parameter fine-tuning, and human preference optimization, enhances the model's capability to generate precise responses, and adapt to the complexities of pediatric medical consultations. 3) Experiments show that PediatricsGPT outperforms other Chinese medical LLMs, validating its effectiveness and reliability in real-world applications. 4) Manual human study is conducted.
Weaknesses
1) Overall I believe the developed model with the curated datasets will be a useful resource. But I think the authors can probably better highlight the novelty in this paper and how the techniques employed in this paper differs from other works. 2) As one of the main contribution of this paper lie in the creation of the dataset, I am not sure if this paper is more suitable for the D&B track in NeurIPS. However, this does not affect my overall rating. 3) The improvements attributed to the universal expert appear marginal, raising questions about whether the gains justify the additional time and memory costs.
Questions
See above.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
Yes
Dear Reviewer jQHJ: We would like to thank the reviewer for taking the time to review our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us. Best regards, Authors
Summary
This paper proposes the first Chinese pediatric Large Language Model (LLM) assistant, designed through a robust training pipeline including continuous pre-training, full-parameter supervised fine-tuning, and direct following preference optimization. The model is shown to outperform existing Chinese medical LLMs in pediatric tasks, and the authors intend to open-source both the model and dataset for community development.
Strengths
- First Chinese Pediatric LLM Assistant with Pediatric Expertise: PediatricsGPT represents a significant milestone as the first Chinese LLM specifically designed to assist with pediatric medical applications, demonstrating expertise in this specialized domain. - Novel Domain Adaptation Paradigm: The paper introduces an innovative training pipeline that includes continuous pre-training, full-parameter supervised fine-tuning, and preference optimization, effectively adapting the model to the pediatric domain.
Weaknesses
- Ethical Issues of the Data: The paper lacks explicit details about the usage agreements for the data sources, including license information and open-source details. Given the data leakage issues inherent in LLMs, clarifying these aspects is crucial for the ethical release and use of both the model and dataset. - Evaluation Datasets: The three main evaluated datasets appear to be private, raising concerns about their accessibility for further research and verification. The paper should clarify whether these datasets can be openly released and if there are any ethic issues of releasing these datasets. - Privacy Concerns with GPT-4 API: The use of the GPT-4 API to build instruction data introduces potential privacy concerns, particularly regarding whether data from guidelines and knowledge bases can be legally and ethically fed into a commercialized API like GPT-4. - Missing Representative Baselines: The paper does not include comparisons with leading medical LLMs such as Meditron, Me-LLaMA, and other advanced models like GPT-4, LLaMA3. Including these baselines would provide a more comprehensive evaluation of PediatricsGPT’s performance.
Questions
N/A
Rating
5
Confidence
4
Soundness
3
Presentation
2
Contribution
3
Limitations
Yes
Dear Reviewer oou4: We would like to thank the reviewer for taking the time to review our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us. Best regards, Authors
Dear Reviewer oou4: As the discussion period is closing, we sincerely look forward to your feedback. We deeply appreciate your valuable time and efforts. It would be very much appreciated if you could once again help review our responses and let us know if these address or partially address your concerns and if our explanations are heading in the right direction. Please also let us know if there are further questions or comments about this paper. We strive to improve the paper consistently, and it is our pleasure to have your feedback! Best regards, Authors
Acknowledgement
Thank you for your response. Most of my concerns have been addressed and I will keep my score.
Response to Reviewer DSSd
Many thanks to the reviewer for the overall constructive and insightful comments!
Response to Reviewer MG71
We thank the reviewer for the meticulous advice!
Thank you for the detailed response. I will keep my score.
Dear Reviewer eZYU: We would like to thank the reviewer for taking the time to review our paper and for the comments. Please kindly let us know if anything is unclear. We truly appreciate this opportunity to clarify our work and shall be most grateful for any feedback you could give to us. Best regards, Authors
Thanks for your rebuttal and I will keep my score.
Dear Author of Paper 385, Thank you for your rebuttal and the additional clarifications provided. After reviewing your comments, the rebuttal document, and revisiting the appendix of your original submission, I still have several concerns regarding your paper. The main issue is not with the model architecture itself. I commend you for thoroughly testing various baseline models on your dataset and proposing a new training pipeline. However, it’s important to note that similar solutions have been proposed for general-domain LLMs. Given that your model and data are positioned as the first Chinese pediatric LLM assistant, the primary contribution of this paper should be the quality of the dataset. Your paper makes a strong assertion that LLMs can be effectively used to answer pediatric questions and generate humanistic responses. However, the conversational responses in the dataset provided do not seem to fully support this claim, as they often appear to be direct extracts from textbooks rather than original, contextually adapted answers. Furthermore, it appears that your model, including the dataset, heavily relies on prior work such as DISC-MedLLM. While the concept of developing an LLM specifically for pediatrics is compelling, the current implementation may not adequately address real-world challenges. I suggest consulting with researchers from different pediatric departments before proceeding further with this dataset to ensure that it meets the practical needs of the field. Additionally, we require more time to discuss the ethical implications of your paper with the ethics reviewers once they submit their evaluations. Therefore, I will be maintaining my current score at this time. However, I will make every effort to reach a consensus with the other reviewers in the ML domain, and it is possible that the final score may be adjusted. Best regards, Reviewer eZYU
Dear Ethical Reviewer: We appreciate your time and effort. The reason we wrote N/A when referring to the IRB in the checklist at the end of the paper is that research involving human subjects in our region of China usually undergoes internal ethical review by the collaborating organization. In other words, we underwent the internal ethical review by the corresponding Chinese healthcare institution, which is fully compliant with the ethical guidelines of NeurIPS 2024. The corresponding guidelines are described below: "In cases when no formal process exists, they can undergo an equivalent informal process (e.g., via their peers or an internal ethics review)."
Hi all, the author-reviewer discussion for this paper is extended to Aug 16 11:59pm ET for authors to reply to ethics reviews. -- PCs
Dear Program Chairs: We looked forward to addressing potential concerns during the extended discussions. However, we have found that the overall discussion seems to be unworkable. We have done our best to respond to comments to avoid misunderstandings and concerns based on multiple responses. We'd like to know if there's anything the authors can do now to. Best regards, Authors
Dear Ethical Reviewer: We look forward to receiving your reply if there is any further clarification required in response to the ethics review. Best regards, Authors
Comment on Ethics Review
Dear Ethics Reviewer pxJq: Thanks for your help taking time to check the ethic issue. Could you reply to the authors? Thanks. It is important for us to give the final score based on the ethic review. And we really hope you can discuss them with the authors. Best, Reviewer eZYU
Response to reviewer eZYU
Dear reviewer eZYU: As the authors are unable to reply under your comment in openreview at this point in time, we are replying to you in the comments under public rebuttal. We appreciate your time and effort as a professional. We very much agree with your discussion and suggestions on the dataset from your professional perspective as a pediatrician. As an effort that seeks to explore and develop both in the pediatric and generic domains, we would like to try to build an initial bridge between the machine learning and pediatric communities by providing a new set of training paradigms for large language models as well as initial data benchmarks. Our original intention is to provide some insights into subsequent research, both at the data and modeling level, through our current efforts. The purpose of doing so is to avoid the serious disconnect you mentioned. We believe that someone has to take the first step in order to drive the future of large language model technology to more valuable applications in real-world clinical practice. We hope that the reviewers will see the valuable and noteworthy contributions of this work, which will be the greatest encouragement for us to continue to work for community development. Best, Authors
Response to reviewer eZYU
Dear reviewer eZYU: Once this work is accepted, the dataset and model will be released immediately. Of course, we can provide the required documents immediately if the policy allows it. We would like to emphasize that the internal ethical review we went through has the authors' names on the approval, and it's possible to violate the double-blind policy. Best, Authors
Official Comment by reviewer eZYU
Thank you for your response. I will consider adjusting the score after further discussion with the ethics reviewer. However, I believe it’s important to think about whether the ML/AI community is truly addressing the needs of the Medical/Pediatrics community. While AI/ML researchers often prioritize model novelty or the introduction of new datasets, those in the medical field are more concerned with addressing real clinical needs. This balance between innovation and practical application deserves careful consideration. Though LLM + X is a very good direction for us to publish papers, datasets and models that are out of touch with actual needs will cause many potential problems.
Response to reviewer eZYU
Dear reviewer eZYU: As the authors are unable to reply under your comment in openreview at this point in time, we are replying to you in the comments under public rebuttal. We appreciate your time and effort as a professional. We very much agree with your discussion and suggestions on the dataset from your professional perspective as a pediatrician. As an effort that seeks to explore and develop both in the pediatric and generic domains, we would like to try to build an initial bridge between the machine learning and pediatric communities by providing a new set of training paradigms for large language models as well as initial data benchmarks. Our original intention is to provide some insights into subsequent research, both at the data and modeling level, through our current efforts. The purpose of doing so is to avoid the serious disconnect you mentioned. We believe that someone has to take the first step in order to drive the future of large language model technology to more valuable applications in real-world clinical practice. We hope that the reviewers will see the valuable and noteworthy contributions of this work, which will be the greatest encouragement for us to continue to work for community development. Best, Authors
Dear Ethics Reviewer pxJq: We have clarified your concerns regarding ethical issues. We would like to know if there is anything else you would like us to clarify so that we can respond as soon as possible. We would like to address these potential concerns before the discussion is concluded to avoid subsequent misunderstandings due to the author's inability to participate in the discussion. We sincerely look forward to your response. Best, Authors
It's reviewer discussion time
Dear all reviewers, This is reviewer discussion time. Your peer reviewers' reviews are in, as well as the author responses. As I understand the process now is when we go in for discussion and aim to form a consensus about this paper. Can I ask each of you to do the following: 1. Read over each other's reviews 2. Review your own review 3. Look over the author's rebuttal and reviewer's feedback. 4. If you feel like any changes or improvements are needed to your own review -- please update accordingly. 5. Post as a comment to confirm that you've taken a look (just so we know who has/hasn't had a chance to respond yet). Thanks!
Discussion with Ethics Reviewer
Dear Ethics Reviewer pxJq: Thanks for your help taking time to check the ethic issue. Could you discuss with us based on authors' reply? Thanks. It is important for us to make the final decision based on the ethic review. Best, Reviewer eZYU
Decision
Accept (poster)