Federated Malware Detection in Flying Ad Hoc Drone Networks using Hybrid Convolutional Learning and Adaptive Optimization

The growing use of drones in edge computing, primarily within Flying Ad Hoc Networks (FANETs), has raised pressing security concerns, particularly in detecting malware within decentralized and privacy-sensitive environments. Traditional intrusion detection systems often fall short in these settings, struggling to meet the demands of limited computational power, non-IID data distributions, and the need to preserve local information. In response to these challenges, we propose F-ResDenseEffiNet, a novel federated deep learning framework that uniquely integrates ResNet, DenseNet, and EfficientNet. This architectural combination, not previously explored in FL-based malware detection, enables residual learning, feature reuse, and compound scaling, resulting in accurate and lightweight detection suitable for UAV platforms. To further improve convergence and adaptability in heterogeneous data settings, we introduce AGRO, an Adaptive Gorilla–Rabbit Optimization algorithm, which balances exploration and exploitation more effectively than standard optimizers such as Adam or RMSProp. Comprehensive experiments on four diverse intrusion detection datasets, DBMD24 (drone-malware), CICIDS2017, NSL-KDD, and KDDCup99 —representing both traditional and UAV-relevant traffic patterns, showcase the model’s strong performance, achieving an average detection accuracy of 97.9%, a threat identification rate of 99.7%, a decision confidence of 98.6%, and an overall classification quality of 98.7%. Additional evaluations, including ablation studies, statistical tests, fairness analysis, and architectural interpretability, further confirm the scalability, reliability, and suitability of the proposed framework for real-time drone-based anomaly detection.