An automated effective communication system in a VR based environment for hearing impaired

In this article, a live speech-to-text generation system has been introduced through automatic speech recognition that will display the caption of the person’s speech to the user and the emotion/intent with which the speaker is speaking. The dataset used for emotional analysis is the RAVDESS (Ryerson Audio-Visual Database of Emotional Speech and Song) dataset. To identify the speaker's emotion, the audio features MFCC(Mel-Frequency Cepstral Coefficients), Mel Spectrogram, and Chroma are extracted. Google's audio recognition module is used for speech-to-text conversion. The emotional analysis of the speech is done through an MLP (Multi-Layer Perceptron) classifier to classify the audio under 4 labels: "Angry", "Sad", "Neutral", and "Happy". The model is evaluated on audio samples from the dataset with basic emotions. The experimental results exhibits reasonable classification rate for emotional difference and the system is validated successfully using the Virtual Reality (VR) based environment.