Brain–Eye Collaboration Target Detection and Localization in Remote Sensing Image

Electroencephalography (EEG) reflects brain mechanisms, and existing research leverages EEG-based brain–computer interfaces for remote sensing image target detection. While most studies focus on achieving target image recognition, there has been less emphasis on target localization, which explores spatial information to enhance detection efficiency. An effective approach for target localization is through eye tracking. In this article, we propose a brain-eye collaboration framework (BECF) that enables both target image recognition and localization. We begin by decoding EEG-based event-related potential (ERP) features using the xDAWN+RG method for target image recognition. Additionally, we implement a region division strategy to achieve both target image recognition and coarse-grained localization, which subsequently guides fine-grained localization. To effectively leverage spatial attention information from the EEG modality and positional information from the EYE modality, we introduced a multimodal network to integrate EEG and EYE modalities into the multimodal messages. Furthermore, we achieve fine-grained target localization through a matching process between multimodal-based potential areas and eye-tracking-based potential areas. For evaluation, we conducted target image detection tasks on our dataset. Notably, the balanced accuracy (BA) of target detection using the division strategy reached 81.32%, with a BA-based information transfer rate of 76.32 bits/min, outperforming other methods and significantly improving detection efficiency.