Adaptive Feature Fusion and Spatio-Temporal Background Modeling in KDE Framework for Object Detection and Shadow Removal

Moving cast shadow identification and extraction are the commonly encountered problems in visual surveillance applications. In the video object detection problem, a shadow is a category of the dynamic entity which needs to be removed for accurate detection of moving objects. In this paper, we have proposed a new scheme to detect the moving object in video frames while removing the shadow. The proposed scheme is based on the notion of background modeling and model learning. The background is modeled by the proposed Spatio-Temporal Kernel Density Estimation (ST-KDE) based model, but the model learning takes place in online mode in the fused feature space. In the fusion process, the Local Binary Pattern (LBP) features of the ST-KDE model of a frame are determined and are fused probabilistically with the Gabor features of the corresponding original frame. To achieve accurate background modeling, weights in the probabilistic fusion are adaptively determined from the scene dynamics. Background model learning is pixel-based where model histograms learn the background with every temporal frame. Learning and classification of the fused feature frames happen simultaneously with frame by frame. The classified frame thus obtained contains the object with some residual shadow which is eliminated by entropy map and thresholding. The main novelties of our paper are: (i) ST-KDE based background modeling, (ii) Determination of the weights of the fusion process in an adaptive framework and, (iii) online background model learning in feature space. The proposed modeling approach has been tested successfully with different video sequences considered from CDnet, Lasiesta, ATON-CVRR, SRD, and ISTD databases. The performance metrics of the proposed scheme have been compared with many existing methods and it has been found that the proposed method exhibits improved performance.