Diffcraft: A Modular Differentiable Framework for Music Synthesis with Timbre Transfer

This paper presents a novel autoencoder-based framework for music synthesis and timbre transfer, addressing limitations of traditional Differentiable Digital Signal Processing (DDSP). By integrating a Variational Autoencoder (VAE) based Audio Encoder with a Mamba-enhanced DDSP Decoder, the framework extracts fundamental frequency ($F_{0}$) and loudness ($L$) directly from waveforms, eliminating reliance on pre-extracted features. The use of Piano Complex Cepstral Coefficients (PCCC) with VAE allowed learning a smooth and rich feature space, enabling flexible audio manipulation. Bidirectional Mamba demonstrated improved performance over the baseline DDSP model. The system achieves high-quality audio generation and efficient timbre transfer. The proposed framework is versatile, offering applications in music transcription, timbre transfer, and multiinstrument synthesis.