A Novel Hybrid Deep Learning Model for Activity Detection Using Wide-Angle Low-Resolution Infrared Array Sensor

In this paper, we propose a deep learning-based technique for activity detection that uses wide-angle low-resolution infrared (IR) array sensors. Alongside with the main challenge which is how to further improve the performance of IR array sensor-based methods for activity detection, throughout this work, we address the following challenges: we employ a wide-angle infrared array sensor with peripheral vision in comparison to a standard IR array sensor. This makes activities at different positions have different patterns of temperature distribution, making it challenging to learn these different patterns. In addition, unlike previous works, our goal is to perform the activity detection using the least possible amount of information. While the conventional works use a time window equal to 10 seconds where a single event occurs, we aim to identify the activity using a time window of less than 1 second. Nevertheless, we aim to improve over the accuracy obtained in previous work by employing deep learning, while keeping the approach light for it to run on devices with low computational power. Therefore, we use a hybrid deep learning model well suited for the classification of distorted images because the neural network learns the features automatically. In our work, we use two IR sensors ( 32×24 pixels), one placed on the wall and one on the ceiling. We collect data simultaneously from both the IR sensors and apply hybrid deep learning classification techniques to classify various activities including “walking”, “standing”, “sitting”, “lying”, “falling”, and the transition between the activities which is referred to as “action change”. This is done in two steps. In the first step, we classify ceiling data and wall data separately as well as the combination of both (ceiling and wall) using a Convolutional Neural Network (CNN). In the second step, the output of the CNN is fed to a Long Short Term Memory (LSTM) with a window size equal to 5 frames to classify the sequence of activities. Through experiments, we show that the classification accuracy of the ceiling data, wall data, and combined data with the LSTM reach 0.96, 0.95, and 0.97, respectively.