Hand Orientation Detection Based on Disparity Maps from Stereo Imagery

Manipulating objects is one of the most basic human interactions. This involves holding, shifting, lifting, and passing objects to other people. These interactions can be replicated for human assistive robot systems in robotics, which can help the elderly with their daily tasks. Many researchers have developed 2D image-based approaches to produce hand landmark detection to recognize hand shapes and gestures. However, there are shortcomings in the resulting data output, namely that it does not provide the characteristics of the distance and orientation of the hand being detected. As a result, it cannot be utilized as input to control the robot manipulator’s movements to get closer to the region of the human hand. We have designed a hand position and orientation recognition system with a stereo camera to anticipate its disparity map, which includes distance prediction data for every map pixel, to overcome this challenge. The disparity map is projected onto a point cloud using a hand landmark detection technique, and the points that match the hand landmarks are chosen. We employ the Singular Value Decomposition (SVD) mathematical technique to ascertain the hand position and orientation. We verified the sequence of tasks completed to determine the position and orientation of the stereo camera. Furthermore, our approach is user-friendly and can be easily applied across a range of simpler systems, achieving a processing speed of approximately 10.03 FPS.