JEDAN: Joint Euclidean Distance and Autoencoder Network for Robust Out-of-Distribution Detection in Radar-Based Hand Gesture Recognition

Detecting Out-of-Distribution (OOD) gestures is vital for reliable radar-based gesture-recognition systems. Traditional autoencoders often fall short in OOD detection because they prioritize minimizing reconstruction error over forming distinct clusters in the latent space. This study proposes Joint Euclidean Distance and Autoencoder Network (JEDAN), a novel approach that enhances OOD detection by integrating autoencoders with a Euclidean distance-based layer. JEDAN learns a compact representation of In-Distribution (ID) gestures while simultaneously shaping the latent space to be more discriminative. This enables the use of both reconstruction error and the proximity of input samples to the learned class centroids for superior OOD detection. Evaluated on a dataset of nine ID and six challenging OOD gestures, JEDAN significantly outperforms existing state-of-the-art OOD detection methods. Notably, it reduces the false positive rate to 5%, compared to 38% for the baseline, and achieves an Area Under the Receiver Operating Characteristic Curve (AUROC) of 99%, compared to 90% for the baseline. To the best of our knowledge, this is the first OOD detection architecture specifically tailored for radar-based gesture recognition that integrates the strengths of Euclidean distance-based layer and reconstruction-based OOD methods in a unified framework, paving the way for more robust real-world applications.