An Industrial Assistance System with Manual Assembly Step Recognition in Virtual Reality

In the era of Industry 4.0, worker assistance systems are becoming more and more important. In order to assist shop floor workers in manual assembly tasks, we implemented an assistance system in virtual reality. A deep neural network was trained to recognize the current work step in real-time during an assembly process, thus giving the assistance system context-awareness. We defined the problem of assembly step recognition as a multivariate time series classification using the poses of the workers head, both hands and all relevant tools and objects. With this definition, the VR environment's output can also be replaced with data from the real world. For our proof-of-concept assembly step recognition system, we created an assembly process consisting of six different work steps, five movable assembly parts and one tool. We showed that we can train an activity recognition model for assembly steps with only 10 assembly recordings. To achieve this, we used multiple data augmentation techniques and proposed a novel method of synthesizing new training data, which we call Path Joining. With only 10 training recordings, we attain a categorical classification accuracy of 81 percent and with 60 recordings we achieve an accuracy of 89 percent.