Neuro-Symbolic Speech Understanding in Aircraft Maintenance Metaverse

In the emerging world of metaverses, it is essential for speech communication systems to be aware of context to interact with virtual assets in the 3D world. This paper proposes the metaverse for aircraft maintenance training and education of Boeing-737, supplied with legacy manuals, 3D models, 3D simulators, and aircraft maintenance knowledge. Furthermore, to navigate and control operational flow in the metaverse, which is strictly followed by maintenance manuals, the context-aware speech understanding module Neuro-Symbolic Speech Executor (NSSE) is presented. Unlike conventional speech recognition methods, NSSE applies Neuro-Symbolic AI, which combines neural networks and traditional symbolic reasoning, to understand users’ requests and reply based on context and aircraft-specific knowledge. NSSE is developed with an industrially flexible approach by applying only synthetic data for training. Nevertheless, the evaluation process performed with various automatic speech recognition metrics on real users’ data showed sustainable results with an average accuracy of 94.7%, Word Error Rate (WER) of 7.5%, and the generalization ability to handle speech requests of users with the non-native pronunciation. The proposed Aircraft Maintenance Metaverse is a cheap and scalable solution for aviation colleges since it replaces expensive physical aircraft with virtual one that can be easily modified and updated. Moreover, the Neuro-Symbolic Speech Executor, playing the role of field expert, provides technical guidance and all the resources to facilitate effective training and education of aircraft maintenance.