STP-Net: Spatio-Temporal Polarization Network for action recognition using polarimetric videos

Deep learning has brought tremendous progress in computer vision and natural language processing, and is used in multiple non-critical applications. A major bottleneck for its use in many other areas is the black box nature of these algorithms, resulting in a lack of explainability in their decisions. One of the key problems identified is the confounding effect, which causes confusion between the desired causes and other irrelevant factors affecting an outcome. This is more pronounced in the spatio-temporal case, such as the bias on the static background in the classification of a video. A way to handle this is by making use of sensors that capture additional scene properties, to mitigate spurious associations. In this work, we integrate the polarimetric videos with deep learning and evaluate it on the popular action recognition problem. We construct a dataset of polarimetric videos for fine-grained actions and study the effect of various parameters, extracted from the polarimetric video frames, as inputs to a deep network. Using these observations, we design a spatio-temporal polarization network (STP-Net) to effectively extract polarimetric features. This is evaluated on the recent HumanAct12 dataset for human activity recognition. Extensive evaluation clearly shows that the polarimetric modality is able to localize the correct action regions, leading to better generalizability.