High Dimensional Vectors based on Discrete Cosine Transform in Generating Background Music from Vocal Sound

Creating background music based on vocal sounds using deep learning is challenging. The task involves multiple input and multiple output cases that require lengthy input and output as data. However, training the models with raw data using low-end computers is expensive since training takes a long time and requires high memory. Previous studies predominantly have used Recurrent Neural Networks or Transformer models trained with the pitch, timing, and amplitude of scripted musical notes to generate new background music sequences. In this research, we proposed a novel training procedure using simple Gated Recurrent Units as the basic architecture of the model, which is trained with compacted high-dimensional vectors based on raw long audio sequences. Our experiments showed promising results, even when executed on low-end hardware. Our best model achieved a cumulative moving average of the loss of around 0.016, and the model successfully produced interesting background music based on vocal sound.