DiReDi: Distillation and Reverse Distillation for AIoT Applications

Artificial Intelligence & Internet of Things (AIoT) have been widely utilized in various application scenarios. Significant efficiency can typically be achieved by deploying different edge-AI models in various real-world scenarios while a few large models manage those edge-AI models remotely from cloud servers. However, customizing edge-AI models for each user's specific application or extending current models to new application scenarios remains a challenge. Inappropriate local training or fine-tuning of edge-AI models by users can lead to model malfunction, potentially resulting in legal issues for the manufacturer. To address the aforementioned issues, this article proposes an innovative framework called “DiReDi”, which involves knowledge Distillation & Reverse Distillation. In the initial step, an edge-AI model is trained with presumed data and a knowledge distillation (KD) process using the cloud AI model in the upper management cloud server. This edge-AI model is then dispatched to edge-AI devices solely for inference in the user's application scenario. When the user needs to update the edge-AI model to better fit the actual scenario, two reverse distillation (RD) processes are employed to extract the knowledge – the difference between user preferences and the manufacturer's presumptions from the edge-AI model using the user's exclusive data. Only the extracted knowledge is reported back to the upper management cloud server to update the cloud AI model, thus protecting user privacy by not using any exclusive data. The updated cloud AI can then update the edge-AI model with the extended knowledge. Simulation results demonstrate that the proposed DiReDi framework allows the manufacturer to update the user model by learning new knowledge from the user's actual scenario with private data. The initial redundant knowledge is reduced since the retraining emphasizes user private data. Furthermore, this model update approach via cloud allows manufacture to check model updates ensuring that all models are managed safely and effectively.