PromptPilot: Controlling UAVs Using Multimodal AI

The integration of Artificial Intelligence (AI) in drone technology has been proven to have a wide range of applications in the logistics, agriculture, construction, security, and military sectors. Furthermore, rapid advancements in large language models and their effectiveness in recent years have been of particular interest in human-robot interaction. Building on these innovations, we present PromptPilot - a multimodal Large Action Model (mLAM) that enables users to interact with drones to execute complex vision-based tasks such as dynamic object tracking using everyday language. We also develop a drone feed classifier that identifies the required modalities of a prompt with a precision of 97.58%, demonstrating strong performance in understanding and executing user queries autonomously.