On-Device Generative AI: The Need, Architectures, and Challenges

The area of Generative Artificial Intelligence (GenAI) is rapidly expanding, as seen by the regular release of new models and applications every few months. While these GenAI models have impressive capabilities, their computational intensity has presented issues, especially in applications demanding low latency. Hence, substantial research is being conducted to develop ways to scale down these models so that they may be used for on-device computing on edge devices. Examining successful examples of GenAI models implemented on mobile devices with minimum latency becomes critical in understanding the practical consequences of these breakthroughs. Notable instances, such as the deployment of diffusion-based GenAI models on flagship smartphones like Samsung S23 Ultra and iPhone 14, demonstrate the possibility and promise of bringing GenAI applications to consumers’ fingertips. We further analyze and find out the approaches and strategies that make these on-device deployments successful.