Information Extraction from Pill Bottle Images via Text Stitching

Extracting information from a sequence of images of a pill bottle captured through a phone camera can allow customers to record useful entities such as drug name, dosage, instructions etc. This information can enable customers to track their medication adherence and interact with a pharmacy to schedule refills though phone applications. However, partial images of pill bottles being text-heavy pose several challenges during image stitching. This is due to the limitations of existing image stitching algorithms, which do not consider text features present in the image. In this work, we propose an end-to-end framework for recognising entities from a sequence of images of pill bottles by using a combination of image and text features. The features are used to generate a text panorama and apply named entity recognition for information extraction. The results obtained on a dataset of pill bottle images show that the accuracy of our proposed framework significantly improves the performance of information extraction from the image sets where state-of-the-art panorama stitching method fails.