Semi-Automated Clinical Lexicon Induction and Its Use in Cohort Selection from Clinical Notes

Special purpose lexicons are invaluable in biomedical natural language processing. They are especially crucial for a task such as the 13-criteria based cohort identification from clinical notes, process in N2C2 2018 Track 1 Challenge. While manually developed lexicons helped us achieve high performance, the process was ad hoc and nonreproducible. This paper presents a semi-automated lexicon induction method, using Logistic Regression (LR) and word embeddings, which brings rigor to the process. The key idea was to use n-grams in the training corpus as features of LR and identify those features (n-grams) with the most impact on the outcome as the lexicon. The semi-automatically generated lexicons achieved overall F measure of 0.9166 versus 0.9003 with manually generated lexicons. Therefore, this study shows that lexicons generated using a rigorous, semi-automated approach can retain performance while bringing rigor to the process.