Musimple: A Simplified Music Generation System with Diffusion Transformer

Music generation has become an important area of research, with applications in automated composition, personalized soundtracks, and virtual entertainment. Recently, diffusion models such as Diffusion Transformers (DiT) have shown promising results in generating high-quality audio. However, many previous music generation systems rely heavily on large datasets and require significant computational resources. This makes them challenging to reproduce and less accessible for broader research. To address these limitations, we propose a simplified text-to-music generation system. Our model uses the small-scale GTZAN dataset and 2D mel spectrograms as the primary audio representation. By avoiding complex components like VAEs, we streamline the architecture and reduce computational overhead. We evaluate our model using metrics such as FAD, KL and CLAP, achieving scores of 5.0, 2.31, and 0.18, respectively. These results demonstrate that our system generates musically acceptable audio even with reduced data. Our code, data, and pre-trained checkpoints are fully open-sourced and available at https://huggingface.co/ZheqiDAI/Musimple.