Multitask Multisource Deep Correlation Filter for Remote Sensing Data Fusion

With the amount of remote sensing data increasing at an extremely fast pace, machine learning-based technique has been shown to perform superiorly in many applications. However, most of the existing methods in the real-time application are based on single modal image data. Although a few approaches use the different source images to represent the object via a fusion scheme, it may not be appropriate for multimodality information processing. In addition, these methods hardly benefit from the end-to-end network training due to the limitations of implementation difficulty and computational cost. In this article, we propose a multitask multisource information fusion method in the deep learning and correlation filter frameworks, which is applied to the fields of tracking and remote sensing data processing. The contribution of individual layers from different source data inside the deep network model is considered as a task. The proposed method can employ interdependencies among different sources data and tasks to learn deep network parameters and filters jointly to improve the performance. Second, we present an effective object appearance selection scheme to adaptively capture the object appearance changes via an effective deep learning network, then integrating information from different modalities to achieve information fusion. Different source information can provide robust performance from different aspects with complementary properties. Third, we further extend the proposed approach to the field of remote sensing for semantic labeling. The layers’ sensitivity is utilized to verify the robustness of different classes. Extensively experiments on five benchmarks show that the proposed approach performs favorably against the state-of-the-arts.