Human Activity Understanding Through Explainable Audio–Visual Features

This letter presents an approach to enhance the well-being and safety of aging populations through better understanding their everyday activities. Our approach uses ambient sensor data and analyzes them with multimodal feature learning and explainable artificial intelligence (XAI) through the ImageBind framework. By integrating the SHapley Additive exPlanations (SHAP) method, our system uncovers intricate patterns within human daily activities. Experimental results reveal significant improvements in activity classification accuracy, particularly with the XGBoost model applied to the Kinetics dataset. Moreover, by utilizing a subset of the most influential features identified through SHAP analysis, our method achieves notable reductions in predictors without sacrificing performance.