Towards Developing Novel and Intuitive Brain-based Control Interface via On-skin Input

Input is a significant challenge for current wear-able systems in the context of extended reality and metaverse environment due to lack of expressive power, naturalness, intuitiveness and social acceptability. In this paper, we propose a novel control interface based on skin touch that exhibits reasonable compatibility with miniaturized wearable systems and alleviates memorability issues by adding a touch dimension to the repertoire of motor and gestural input patterns. This onskin touch-motion interface allows the body’s surface to be annexed as an input surface of the wearable system with limited input surface, without the need for the skin to be invasively instrumented with sensors, tracking markers, or other on-skin devices. We designed a deep learning-based method to decode the user’s intentional skin-touch using multi-scale temporal and spatial convolution modules for efficient extraction of touch and motor-related features from our challenging dataset, along with a conditional affine transformation module for capturing the inter-relationship between touch and motion features. The proposed method achieved an average classification accuracy of 57.82 (±1.79) % for the touch-motion four classes and 48.12 (±2.05) % for the motion four classes across twelve subjects. Our end-to-end model shows superior performance compared to conventional deep neural networks in terms of classification performance, class confusion, and feature discriminability, but with fewer parameters, demonstrating considerable potential for real-life applications.