Harnessing Knowledge-Distillation for Lightweight AI-Implementation on Resource-Constrained Device

Smart wearable devices benefit from lightweight deep neural networks (DNNs) with small silicon footprints for efficient real-time processing. While compression techniques are often applied to heavier models, achieving lightweight DNNs for wearable technology, particularly in real-time applications, has been underexplored. This work proposes a knowledge-distillation approach to derive a lightweight student model from a pre-trained teacher model, which maintains high performance. Tested on real-world ECG data, the student DNN achieves 89% classification accuracy, matching the teacher model, with a 93.12% memory reduction. Furthermore, deployment on resource-constrained devices like Coral TPU shows a 2 to 6.5-fold throughput improvement compared to other devices.