Action Localization and Recognition through Unsupervised I3D and TSN

Since CCTV technology was widely introduced as a video surveillance system, it has become very popular due to increased security concerns. However, detecting suspicious behavior is still done manually by humans, which is inefficient in terms of accuracy and manpower (prone to distraction and fatigue). Many studies use the entire frame of a video to recognize human action, whereas in this study first localize the area of object movement using the unsupervised inflated 3D (I3D) network which produces several frames that have been determined as the region of interest (ROI) from the total frames in the video. Then the collection of ROI frames is generated into a video which will then be classified to recognize human action through the Temporal Segment Networks (TSN). For efficiency purposes, the proposed pipeline can be run separately in the edge (for localization), and cloud environment (for recognition), respectively. The results show that there are 54 classes of ROI video data sets that have an average human action recognition prediction similarity rate above 84.6% which is the average of the prediction similarity rate of the entire ROI video data. In terms of prediction accuracy, 37 classes of data (36.63%) are better than the original class, including 25 classes of data (24.75%) above the threshold, and the processing speed of classification with ROI video data is on average 5 times faster than full-frame video data.