Uncertainty-Aware Topological Persistence Guided Knowledge Distillation on Wearable Sensor Data

In applications involving analysis of wearable sensor data, machine learning techniques that use features from topological data analysis (TDA) have demonstrated remarkable performance. Persistence images (PIs) generated through TDA prove effective in capturing robust features, especially to signal perturbations, thus complementing classical time-series features. Despite its promising performance, utilizing TDA to create PI entails significant computational resources and time, posing challenges for applications on small devices. Knowledge distillation (KD) emerges as a solution to address these challenges, as it can produce a compact model. Using multiple teachers one trained with raw time-series and another with topological features, is a viable approach to distill a single compact student model. In such a case, the two teachers will have different statistical characteristics and need some form of feature harmonization. To tackle these issues, we propose uncertainty-aware topological persistence guided knowledge distillation. This approach involves separating common and distinct components between teachers and applying varying weights to control their effects. To enhance the knowledge provided to a student, uncertain features from teachers are rectified using uncertainty scores. We leverage feature similarities to offer more valuable information and employ relationships computed based on orthogonal properties to prevent excessive feature transformation. Ultimately, our method yields a robust single student that operates solely on time-series data at test-time. We validate the effectiveness of the proposed approach through empirical evaluations across various combinations of models and datasets, demonstrating its robustness and efficacy in different scenarios. The proposed method enhances the classification performance of a student model by approximately 4.3% compared to a model learned from scratch on GENEActiv.