Developing a Multimodal System for Bee Object Detection and Health Assessment

Honey bees pollinate about one-third of the world’s food supply, but bee colonies have alarmingly declined by nearly 40% over the past decade due to several factors, including exposure to pesticides. Traditional methods for monitoring beehives, such as human inspections, are subjective, disruptive, and time-consuming. To address these limitations, artificial intelligence has been used to assess beehive health. However, previous studies have lacked an end-to-end solution and primarily relied on data from a single source, either bee images or sounds. This study introduces a unified system for bee object detection, which selectively processes data containing bees for health evaluation. Additionally, it utilizes a combination of visual and audio signals to analyze bee behaviors. A Cross-Attention-based Multimodal Neural Network (CAMNN) was developed to adaptively focus on key features from each type of signal for more accurate bee health assessment. The CAMNN achieved an overall accuracy of 93%, surpassing eight existing single-signal Convolutional Neural Networks and Recurrent Neural Networks. It outperformed the best image-based model by 33% and the top audio-based model by 15%. Furthermore, it improved prediction robustness, attaining an F1-score above 90% across all four evaluated health conditions. The study also showed that audio signals were more reliable than images for assessing bee health. By seamlessly integrating CAMNN with both image and sound data in a holistic bee health monitoring system, this approach provided a more efficient and non-invasive solution for the early detection of bee diseases and the preservation of bee colonies.