Decision Making Engine for Task Offloading in On-device Inference Based Mobile Applications

On-device AI is one of the latest cutting-edge technologies which allows devices to run the machine learning models on the device. This provides low latency, fewer privacy concerns, and many other advantages. But even though modern mobile devices come with great performance, there are situations where these mobile devices cannot handle the resource requirements of these on-device based mobile applications. As a solution for this lack of resources issue, mobile devices can transfer their heavy computational tasks to other devices such as nearby devices to minimize the burden. Transferring all computational tasks to resource-rich devices will not yield favorable under all circumstances. Depending on various conditions such as network strength and execution time, the mobile device should be able to choose a way to execute the task, either execute locally or execute remotely with the help of a resource-rich device. This paper will discuss the implementation of a decision making engine that decides when to offload machine learning tasks in on-device inference based Android mobile applications. Main objective of the proposed solution is to minimize the execution delay. Experimental results show that our decision making engine has an accuracy of 81.33% in identifying the best decision. The proposed research is a novel approach designed for on-device inference based Android mobile applications.