Surface Automation - Interacting with Applications using Black Box Approach

One of the biggest challenges of any Robotic Process Automation (RPA) tools is automating scenarios in tricky environments such as virtual machines (VMs) and remote desktops or when dealing with legacy applications where the technical know-how of the interfaces are limited. Typical approach to interacting with applications under regular scenarios is to use application technology specific APIs to build knowledge metadata. However, these approaches fail when it comes to automating in the black box environments because we cannot have technical understanding or APIs of the applications. Now, the only way to automate here is by taking the screenshot of the applications and understanding the controls available inside the screenshots using image processing algorithms. In this paper we present a novel approach for an end-to-end automation of such scenarios. We employ various AI routines to understand structural and semantic aspects of enterprise applications which in turn are used for building automations and we group all these techniques under a common umbrella of Surface Automation. Our approach involves using robust Optical Character Recognition (OCR) which internally uses LSTMs for reading texts like labels, Natural Language Processing (NLP) for semantic understanding of the page, Image Region Similarity for matching icons like home button and Computer Vision based Object Detection for recognizing predefined controls [text-fields, buttons, dropdown etc.].