Development of a virtual coach scenario for hand therapy using LEAP motion

Hand therapy is the art that fills the gap between surgery and practical life. It helps the patient to regain the hands full functionality after a certain injury or surgery. Hand therapy could be a very tedious process that implies physical exhaustion. Also finding appointments with the therapist frequent enough for an efficient healing process, is difficult and costly. Since trying new technologies is usually exciting to people, using the advancements in the field of artificial intelligence could be a solution to this. A virtual therapist (VT) was implemented to help the patient do his exercises at home in an engaging gamified environment. The VT artificial intelligence used hierarchical finite state machine architecture. This VT can explain the exercise, monitor the patient, correct error and assess the performance of the patient during the exercise. To test the efficiency of the VT, a web hand therapy exercise was implemented using Unity platform to build the exercise environment. LEAP motion technology was used to detect the information of the hand movement. The exercise was chosen to be run on the web to enable its access to the user from home and allow the therapist to have access to the data of the patient. Facial expression detection was used to keep track of the patients' facial expression to detect whenever he feels pain. At the end of the exercise, a general feedback is given to the patient to keep track of his progress. This exercise was tested with 19 persons. The idea of being coached by a VT was very welcomed by the test subjects. The exercise was fun and motivating to them. The VT guidance and assessment were very helpful and easy to follow. However, some modifications are needed in the pain detection part to form a more efficient exercise.