Handwriting Recognition for Medical Prescriptions using a CNN-Bi-LSTM Model

It is commonly seen that it is tough to read the handwritten text from medical prescriptions. It is mostly due to the different style of handwriting and the use of Latin abbreviations for medical terms which is usually unknown to the general public. This can make it difficult for both patients and even pharmacists to read the prescription, which can have negative or even fatal consequences if read incorrectly. This paper demonstrates the use of a CNN-Bi-LSTM model along with Connectionist Temporal Classification. The prescribed model consists of three components, the convolutional layers for feature extraction, the Bi-LSTM network for making predictions for each frame of the context vector and the final decoding to translate each character in the recognized sequence by LSTM layers into an alphabetic character using the CTC loss function. A linear layer is added after the bi-LSTM layer to compute the final probabilities, which will be decoded. We also built a corpus manually containing the terms widely used in the medical domain, commonly used in prescriptions. We then use string matching algorithms, and string distance functions to find the nearest word in the corpus, so that bias is given to medical terms for increasing accuracy of the predicted output.