Abstract:
Deep learning associated with neurological signals is poised to drive major advancements in diverse fields such as medical diagnostics, neurorehabilitation, and brain-com...Show MoreMetadata
Abstract:
Deep learning associated with neurological signals is poised to drive major advancements in diverse fields such as medical diagnostics, neurorehabilitation, and brain-computer interfaces. The challenge in harnessing the full potential of these signals lies in the dependency on extensive, high-quality annotated data, which is often scarce and expensive to acquire, requiring specialized infrastructure and domain expertise. To address the appetite for data in deep learning, we present Neuro-BERT, a self-supervised pre-training framework of neurological signals based on masked autoencoding in the Fourier domain. The intuition behind our approach is simple: frequency and phase distribution of neurological signals can reveal intricate neurological activities. We propose a novel pre-training task dubbed Fourier Inversion Prediction (FIP), which randomly masks out a portion of the input signal and then predicts the missing information using the Fourier inversion theorem. Pre-trained models can be potentially used for various downstream tasks such as sleep stage classification and gesture recognition. Unlike contrastive-based methods, which strongly rely on carefully hand-crafted augmentations and siamese structure, our approach works reasonably well with a simple transformer encoder with no augmentation requirements. By evaluating our method on several benchmark datasets, we show that Neuro-BERT improves downstream neurological-related tasks by a large margin.
Published in: IEEE Journal of Biomedical and Health Informatics ( Early Access )
Funding Agency:
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Autoencoder ,
- Deep Learning ,
- Inverse Fourier Transform ,
- Gesture Recognition ,
- Fourier Domain ,
- Transformer Encoder ,
- Neurological Activity ,
- Deep Learning Data ,
- Classification Task ,
- Data Augmentation ,
- Autoregressive Model ,
- Target Prediction ,
- Representation Learning ,
- EEG Recordings ,
- Random Initialization ,
- Mean Average Precision ,
- Self-supervised Learning ,
- Masked Images ,
- Linearly Separable ,
- Top-1 Accuracy ,
- Prediction Head ,
- Vision Transformer ,
- Hand Gesture Recognition ,
- Inverse Discrete Fourier Transform ,
- Linear Decoder ,
- Pre-training Method ,
- Pretext Task ,
- Sequence Embedding ,
- Random Weight Initialization ,
- Original Signal
- Author Keywords
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Autoencoder ,
- Deep Learning ,
- Inverse Fourier Transform ,
- Gesture Recognition ,
- Fourier Domain ,
- Transformer Encoder ,
- Neurological Activity ,
- Deep Learning Data ,
- Classification Task ,
- Data Augmentation ,
- Autoregressive Model ,
- Target Prediction ,
- Representation Learning ,
- EEG Recordings ,
- Random Initialization ,
- Mean Average Precision ,
- Self-supervised Learning ,
- Masked Images ,
- Linearly Separable ,
- Top-1 Accuracy ,
- Prediction Head ,
- Vision Transformer ,
- Hand Gesture Recognition ,
- Inverse Discrete Fourier Transform ,
- Linear Decoder ,
- Pre-training Method ,
- Pretext Task ,
- Sequence Embedding ,
- Random Weight Initialization ,
- Original Signal
- Author Keywords