Improving Translation from Portuguese to Brazilian Sign Language with Speech-to-Text Integration Through Natural Language Processing

This innovative practice full paper focuses on developing a Portuguese-to-Libras translation model designed to facilitate the participation of deaf students in academic conferences in the absence of sign language interpreters. The pursuit of remote solutions, such as virtual translators for simultaneous interpretation from Portuguese to Libras, is intended to facilitate the inclusion of deaf students in higher education during the pandemic, despite the presence of bureaucratic barriers and interpreter shortages. The education of deaf individuals presents a distinctive set of challenges that necessitate the development of innovative technological and methodological approaches to enhance inclusion and promote academic success. Some studies have investigated these issues and put forward proposals for improving the educational experience of deaf individuals. The overarching objective of the research was to apply natural language processing (NLP) techniques, utilizing the Transformer deep learning neural network architecture, to translate Brazilian Portuguese into Libras, thereby facilitating real-time translation and emulating the actions of human interpreters through a virtual character. The specific objectives were to create a dataset comprising Portuguese sentences and their corresponding Libras representations, develop a speech-to-text algorithm, identify and adapt suitable models for translation, and evaluate the performance of translations and the integration of speech-to-text with the NLP model. These objectives were established to ensure accuracy and efficacy in translation and system integration. This research project entailed the construction of an automated translation model from Portuguese to Libras, based on the T5 architecture from Hugging Face. Additionally, a speech recognition system was developed to transcribe audio into text. The translation accuracy, evaluated using the BLEU metric, was 75.32%, which was compared with the glossed sentences in Libras used as the linguistic corpus. However, the model encountered challenges with diacritics and specific vocabulary. The simultaneous interpreter, which combined speech-to-text with the NLP model, demonstrated promising results in real-time translations to Libras.