FSOS-AMC: Few-Shot Open-Set Learning for Automatic Modulation Classification Over Multipath Fading Channels

Automatic modulation classification (AMC) plays a vital role in advancing future wireless communication networks. Although deep learning (DL)-based AMC frameworks have demonstrated remarkable classification capabilities, they typically require large-scale training datasets and assume consistent class distributions between training and testing data-prerequisites that prove challenging in few-shot and open-set scenarios. To address these limitations, we propose a novel few-shot open-set AMC (FSOS-AMC) framework that integrates a multisequence multiscale attention network (MS-MSANet), meta-prototype training, and a modular open-set classifier. The MS-MSANet extracts features from multisequence input signals, while meta-prototype training optimizes both the feature extractor and the modular open-set classifier, which can effectively categorize testing data into known modulation types or identify potential unknown modulations. Extensive simulation results demonstrate that our FSOS-AMC framework achieves superior performance in few-shot open-set scenarios compared to state-of-the-art methods. Specifically, the framework exhibits higher classification accuracy for both known and unknown modulations, as validated by improved accuracy and area under the receiver operating characteristic curve (AUROC) metrics. Moreover, the proposed framework demonstrates remarkable robustness under challenging low signal-to-noise ratio (SNR) conditions, significantly outperforming existing approaches.