Free text mining of TCM medical records based on conditional random fields

Medical records of Traditional Chinese Medicine (TCM) are usually free text and unstructured data, how to extract medical terms from TCM medical records based on conditional random fields is an interesting problem. TCM medical records obtained from dermatology in Guangdong Provincial Hospital of Chinese Medicine are segmented to single words and labeled with grammatical properties of words by TCM expert, and divided into training sets and testing sets. Clinical terms of training sets are also labeled with clinical terms labeling. In order to evaluate the recognition ability of CRF, three indicators (recognition Precision (P), recognition Recall (R) and F-score (F)) are defined, and three comparisons are given: CRF compared with HMM, ME, and MEMM; Cross test with CRF; and recognition accuracies on different types of clinical terms with CRF. The result shows that, CRF performs a best recognition ability comparing with HMM, ME and MEMM in different groups with different levels of training sets. CRF is credible and stable for different training samples by ten cross test. With the same CRF model, terms of TCM diagnosis can be identified most easily, and terms of symptoms and signs are relatively difficult to identify. So CRF is a suitable model on mining clinical terms in free text of TCM medical records.