LSTM and Chroma Feature Based Music Composer

Several research papers showed that long short-term memory (LSTM) recurrent neural networks (RNNs) are a powerful tool in algorithmic composing. They can learn both fine (short-term) and coarse (long-term) structures in music. Most of the promising results were got by using musical notation as input data. Can LSTMs give similar results by using time-series data, such as audio? This research performs a baseline experiment of an LSTM composing algorithm using audio data as input instead of musical notation. The audio data is presented to the LSTM as a chroma feature encoded to an integer representation. The LSTM is trained on a single piano piece with a segment held out. After, it is tasked with generating a song of a desired duration based on the held out segment. The experimental results show the LSTM has learned the coarse structure of the audio data (the key), but has not learned the fine structure (the melody). Because of this and degrading the audio quality when reconstructing from chroma features, the resulting output does not sound like music. The authors propose the results show only a proof of concept for the algorithm and that more work is needed.