Multimodal Data Fusion Method for Music Education Evaluation: Comprehensive Application of Image Processing and Audio Analysis

The traditional music assessment method is limited to a single data source, and it is difficult to evaluate students’ music performance comprehensively. To solve this problem, this study adopted the method of multi-modal data fusion, combining audio and image data for evaluation, to provide a more comprehensive music education evaluation method. Specific steps include: firstly, multimodal data was collected, including audio and images, and the collected data was preprocessed for denoising, standardization, and alignment. Secondly, relevant features were extracted from audio and image data. Then, convolutional neural network (CNN) was used to process image data; recurrent neural network (RNN) was used to process audio data, and their outputs were connected together. Multimodal Transformers were used to construct evaluation models, which were predicted and analyzed based on the input of multimodal data. Finally, the accuracy and effectiveness of the model were evaluated and optimized. The experimental results showed that the accuracy of the model optimized by hyperparameters reached 0.99, and the proposed method could effectively improve the quality of music education evaluation and comprehensively reflect students’ music performance.