SiBo-The Sign Bot, Connected World for Disabled

Sign Language has proven to be a leap forward in enabling a communication medium for mute and hard-of-hearing people, yet it is not a recognized language among others, making the challenged ones feel socially excluded. In this paper, the proposed assistive device - SiBo, brings the power of embedded AI to a subset of people, often sidelined when delivering mainstream technology. This proposed device exploits the capabilities of Infineon's Sense2GoL radar and its MCU portfolio in interpreting and recognizing the visual-manual modality of communication - American Sign Language (ASL) to control the smart home components. This work explores the state-of-the-art radar technology in gesture recognition and proposes an edge deployable deep neural network model with a small memory footprint. The developed model extensively uses ML pipeline services offered by Edge Impulse enabling faster prototyping methods for such edge applications. It is capable of classifying the input ASL gestures in real-time with an accuracy of 89.4% and inference latency of 2 ms. The already existing methods utilize high-end cameras to capture hand gestures. But, they being susceptible to adverse weather conditions and posing an added risk of privacy breaches or data vulnerabilities makes it less favorable in such applications and hence gives an edge to radar technology.