A Traditional Chinese Medicine large language model, inspired by the wisdom of the eminent representative of ancient Chinese medical scholars, Zhang Zhongjing.
A Traditional Chinese Medicine large language model, inspired by the wisdom of the eminent representative of ancient Chinese medical scholars, Zhang Zhongjing.
This model aims to illuminate the profound knowledge of Traditional Chinese Medicine, bridging the gap between ancient wisdom and modern technology, and providing a reliable and professional tool for the Traditional Chinese Medical fields. However, all generated results are for reference only and should be provided by experienced professionals for diagnosis and treatment results and suggestions.
This model aims to illuminate the profound knowledge of Traditional Chinese Medicine, bridging the gap between ancient wisdom and modern technology, and providing a reliable and professional tool for the Traditional Chinese Medical fields. However, all generated results are for reference only and should be provided by experienced professionals for diagnosis and treatment results and suggestions.
<palign="center"><b>Fig 1. A logo of CMLM-ZhongJing generated by Bing’s drawing output combined with human creative prompts.</b></p>
<palign="center"><b>Fig 1. A logo of CMLM-ZhongJing generated by Bing’s drawing output combined with human creative prompts.</b></p>
## 1.Instruction Data Construction
## 1.Instruction Data Construction
While many works such as Alpaca, Belle, etc., are based on the self-instruct approach which effectively harnesses the knowledge of large language models to generate diverse and creative instructions, this approach may lead to noise in instruction data, thereby affecting the accuracy of the model in fields where professional knowledge has a low tolerance for errors, such as medical and legal scenarios. Therefore, how to quickly invoke the OpenAI API without sacrificing the professionalism of instruction data has become an important research direction for instruction data construction and annotation scenarios. Here, we will briefly describe our preliminary experimental exploration.
While many works such as Alpaca, Belle, etc., are based on the self-instruct approach which effectively harnesses the knowledge of large language models to generate diverse and creative instructions, this approach may lead to noise in instruction data, thereby affecting the accuracy of the model in fields where professional knowledge has a low tolerance for errors, such as medical and legal scenarios. Therefore, how to quickly invoke the OpenAI API without sacrificing the professionalism of instruction data has become an important research direction for instruction data construction and annotation scenarios. Here, we will briefly describe our preliminary experimental exploration.
<palign="center"><b>Fig 2. A Multi-task Therapeutic Behavior Decomposition Instruction Construction Strategy in the Loop of Human Physicians.</b></p>
<palign="center"><b>Fig 2. A Multi-task Therapeutic Behavior Decomposition Instruction Construction Strategy in the Loop of Human Physicians.</b></p>
...
@@ -21,8 +17,6 @@ While many works such as Alpaca, Belle, etc., are based on the self-instruct app
...
@@ -21,8 +17,6 @@ While many works such as Alpaca, Belle, etc., are based on the self-instruct app
#### 1.1 Multi-task Therapeutic Behavior Decomposition Instruction Construction Strategy
#### 1.1 Multi-task Therapeutic Behavior Decomposition Instruction Construction Strategy
Human memory and understanding require the construction of various scenarios and stories to implicitly encode knowledge information. The clarity of memory depends on the duration and richness of the learning process. Interleaved learning, spaced practice, and diversified learning can enhance the consolidation of knowledge, thereby forming a deep understanding of domain knowledge. Our approach is to learn from the process of human memory knowledge, use professional tables, leverage the language representation capabilities of large language models, strictly set specific prompt templates, so that the model can generate 15 scenarios based on the table data of Chinese medicine gynecology prescriptions, including patient therapeutic story, diagnostic analysis, diagnosis treatment expected result, formula function, interactive story, patient therapeutic story, narrative medicine, tongue & pulse, therapeutic template making, critical thinking, follow up, prescription, herb dosage, case study, real-world problem, disease mechanism, etc., to promote the model's reasoning ability for prescription data and diagnostic thinking logic.
Human memory and understanding require the construction of various scenarios and stories to implicitly encode knowledge information. The clarity of memory depends on the duration and richness of the learning process. Interleaved learning, spaced practice, and diversified learning can enhance the consolidation of knowledge, thereby forming a deep understanding of domain knowledge. Our approach is to learn from the process of human memory knowledge, use professional tables, leverage the language representation capabilities of large language models, strictly set specific prompt templates, so that the model can generate 15 scenarios based on the table data of Chinese medicine gynecology prescriptions, including patient therapeutic story, diagnostic analysis, diagnosis treatment expected result, formula function, interactive story, patient therapeutic story, narrative medicine, tongue & pulse, therapeutic template making, critical thinking, follow up, prescription, herb dosage, case study, real-world problem, disease mechanism, etc., to promote the model's reasoning ability for prescription data and diagnostic thinking logic.
@@ -34,8 +28,7 @@ Human memory and understanding require the construction of various scenarios and
...
@@ -34,8 +28,7 @@ Human memory and understanding require the construction of various scenarios and
#### 1.2 Regular TCM Instruction Data Construction Strategy
#### 1.2 Regular TCM Instruction Data Construction Strategy
In addition, we have also added instructions based on the content of Chinese medicine ancient books, noun explanations, symptom synonyms, antonyms, syndromes, symptoms, treatment methods, etc. In order to form a control experiment, we only use one instruction template to represent data for this part, and the number of this part of the data is 80,000, which is significantly more than the number of instructions constructed by the above strategy. The following is the specific number of instructions and tokens information.
In addition, we have also added instructions based on the content of Chinese medicine ancient books, noun explanations, symptom synonyms, antonyms, syndromes, symptoms, treatment methods, etc. In order to form a control experiment, we only use one instruction template to represent data for this part, and the number of this part of the data is 80,000, which is significantly more than the number of instructions constructed by the above strategy. The following is the specific number of instructions and tokens information.
@@ -75,8 +68,6 @@ Data Source and Instruction Quantity Table:
...
@@ -75,8 +68,6 @@ Data Source and Instruction Quantity Table:
## 2.Model Performance Comparison
## 2.Model Performance Comparison
Our test data are based on real medical cases from highly skilled traditional Chinese medicine doctors, typically case reports from provincial renowned senior traditional Chinese medicine practitioners or national medical master level. This kind of data, which is strictly considered as out-of-distribution data (both in terms of subject matter and training dataset distribution, distinct from traditional training and validation sets), is used to ensure a degree of professionalism. In preliminary comparisons with large language models such as Wenxin Yiyan and Spark, we found that our model exhibits good generalization capabilities on a diversified therapeutic decomposition instruction dataset constructed based on 300 Traditional Chinese Medicine prescription data. This perhaps initially confirms that, like humans, large language models are more conducive to learning metaphorical knowledge and logic from text content represented in diverse forms.
Our test data are based on real medical cases from highly skilled traditional Chinese medicine doctors, typically case reports from provincial renowned senior traditional Chinese medicine practitioners or national medical master level. This kind of data, which is strictly considered as out-of-distribution data (both in terms of subject matter and training dataset distribution, distinct from traditional training and validation sets), is used to ensure a degree of professionalism. In preliminary comparisons with large language models such as Wenxin Yiyan and Spark, we found that our model exhibits good generalization capabilities on a diversified therapeutic decomposition instruction dataset constructed based on 300 Traditional Chinese Medicine prescription data. This perhaps initially confirms that, like humans, large language models are more conducive to learning metaphorical knowledge and logic from text content represented in diverse forms.
@@ -93,36 +84,23 @@ Our test data are based on real medical cases from highly skilled traditional Ch
...
@@ -93,36 +84,23 @@ Our test data are based on real medical cases from highly skilled traditional Ch
Our preliminary tests reveal that the ZhongJing large language model demonstrates a certain degree of diagnostic and prescription capabilities not only in gynecology but also in other clinical specialties of traditional Chinese medicine, indicating its potential for generalization. This finding is significant as it suggests that our approach of using a multi-task therapeutic decomposition strategy and a domain-specific million-level instruct data set is effective in enhancing the model's reasoning ability for prescription data and diagnostic thinking logic. It also indicates the potential of large language models (7B parameters level) in fields where professional knowledge has a low tolerance for errors, such as medical and legal scenarios.
Our preliminary tests reveal that the ZhongJing large language model demonstrates a certain degree of diagnostic and prescription capabilities not only in gynecology but also in other clinical specialties of traditional Chinese medicine, indicating its potential for generalization. This finding is significant as it suggests that our approach of using a multi-task therapeutic decomposition strategy and a domain-specific million-level instruct data set is effective in enhancing the model's reasoning ability for prescription data and diagnostic thinking logic. It also indicates the potential of large language models (7B parameters level) in fields where professional knowledge has a low tolerance for errors, such as medical and legal scenarios.
- [ ] Adopt a multi-task therapeutic decomposition strategy, based on multidisciplinary data such as internal medicine, gynecology, pediatrics, and orthopedics, to fine-tune the model with a domain-specific million-level instruct data.
- [ ] Adopt a multi-task therapeutic decomposition strategy, based on multidisciplinary data such as internal medicine, gynecology, pediatrics, and orthopedics, to fine-tune the model with a domain-specific million-level instruct data.
- [ ] Continuously iterate and update. Subsequent releases will include Li Shizhen, Wang Shuhe, Huangfu Mi, Sun Simiao, Ge Hong, and Qihuang version of the large language model for Traditional Chinese Medicine.
- [ ] Continuously iterate and update. Subsequent releases will include Li Shizhen, Wang Shuhe, Huangfu Mi, Sun Simiao, Ge Hong, and Qihuang version of the large language model for Traditional Chinese Medicine.
The Lora fine-tuning part of this project draws on the ideas of alpaca-lora and Chinese-Vicuna. We would like to express our gratitude to the members of the relevant research teams.
The Lora fine-tuning part of this project draws on the ideas of alpaca-lora and Chinese-Vicuna. We would like to express our gratitude to the members of the relevant research teams.
This research is for academic research use only, commercial use is not allowed without permission, and it is not to be used in medical scenarios or scenarios with potential medical intent for clinical practice. This large language model for Traditional Chinese Medicine is still in the laboratory testing stage. The emerging syndrome classification and prescription generation capabilities at this stage are still rudimentary, and it does not yet have a highly reliable clinical diagnostic and therapeutic capability for gynecology and other clinical specialties. The output results are for internal reference testing only. Real medical diagnosis and decision-making still need to be issued by experienced physicians through a strictly regulated diagnostic and therapeutic process.
This research is for academic research use only, commercial use is not allowed without permission, and it is not to be used in medical scenarios or scenarios with potential medical intent for clinical practice. This large language model for Traditional Chinese Medicine is still in the laboratory testing stage. The emerging syndrome classification and prescription generation capabilities at this stage are still rudimentary, and it does not yet have a highly reliable clinical diagnostic and therapeutic capability for gynecology and other clinical specialties. The output results are for internal reference testing only. Real medical diagnosis and decision-making still need to be issued by experienced physicians through a strictly regulated diagnostic and therapeutic process.
Data processing and annotation is one of the important steps in training the model. We sincerely welcome Traditional Chinese Medicine practitioners with strong TCM thinking and innovative spirit to join us. We will also declare corresponding data contributions. We look forward to the day when we can achieve a reliable General Artificial Intelligence for Traditional Chinese Medicine, allowing the ancient Chinese medicine to blend with modern technology and shine anew. This is also the ultimate mission of this project. If interested, please send an email to 21110860035@m.fudan.edu.cn.
Data processing and annotation is one of the important steps in training the model. We sincerely welcome Traditional Chinese Medicine practitioners with strong TCM thinking and innovative spirit to join us. We will also declare corresponding data contributions. We look forward to the day when we can achieve a reliable General Artificial Intelligence for Traditional Chinese Medicine, allowing the ancient Chinese medicine to blend with modern technology and shine anew. This is also the ultimate mission of this project. If interested, please send an email to 21110860035@m.fudan.edu.cn.
This project is jointly guided by Professor Zhang Wenqiang from Fudan University and Professor Wang Haofen from Tongji University. It is completed by Kang Yanlan, Chang Yang, and Fu Jiyuan, members of the [ROI Lab](https://www.fudanroilab.com/) at Fudan University.
This project is jointly guided by Professor Zhang Wenqiang from Fudan University and Professor Wang Haofen from Tongji University. It is completed by Kang Yanlan, Chang Yang, and Fu Jiyuan, members of the [ROI Lab](https://www.fudanroilab.com/) at Fudan University.
