Joint and transfer learning of CNN and similarity metric for comparing UAV video image patches

Comparing Unmanned Aerial Vehicle (UAV) video image patches is a fundamental task in UAV video image processing. The main difficulty lies in the wide variety of appearance changes in images taken under different UAV imaging conditions, such as jitter, changing points of view, frequent undefined motion, illumination changes and etc. The usual algorithms which are based on the hand-craft features and independently predefined similarity metrics, cannot deal with these factors well. Motivated by recent successes on learning deep feature representations and feature similarity metric, a method which jointly models and learns these two objects is proposed here. Especially, comparing UAV video image patches is deemed as a binary classification problem and a Convolutional Neural Network (CNN) based comparing system is developed. It is composed of three parts: (1) two stream CNNs, (2) one similarity metric network, (3) one softmax layer. To jointly learn the CNNs and similarity metric, the available standard natural image datasets are employed and two new datasets representing typical satellite and UAV imaging scenes are built. Furthermore, over the datasets from different imaging scenes, the transfer joint learning of the proposed comparing system is investigated. The primary experimental results show that the proposed method can significantly outperform the recent results of the hand-craft feature based comparing methods.