Extraction and Classification of Medicines from Handwritten Medical Prescriptions

In the modern world of rapid technological progress and digital transformation, industries are swiftly shifting from paper-based to digital systems. Document digitization, especially for image-based documents lacking machine readability, is gaining global attention. Sectors like real estate, finance, law, healthcare, and education are actively adopting digital strategies. In healthcare, where handwritten prescriptions lead to inefficiencies such as errors in medicine name by pharmacist due to doctor's hand writing lead to catastrophic failure and delays in various processes. Therefore, digital solutions are being developed to enhance patient care. This study proposes an approach to extract and categorize medicines from handwritten prescriptions, improving accuracy and communication with pharmacists. We proposed an approach that uses stroke enhancement apart from pre-processing with comprehensive multi-modal architecture, including a transformer based deep neural network and a sequence-to-sequence model such as DONUT and layoutLMv2 model. Our approach achieves a mean accuracy of about 0.85%. It surpasses models like LayoutLMv2 and DONUT by providing a balance in accuracy and consistency. Furthermore, the extracted features are passed to the various state of art machine learning classification algorithms upon which Gradient Boosting, Random forest and Ridge classifier produced the promising results for medicine type classification.