Automatic Multitrack Mixing With A Differentiable Mixing Console Of Neural Audio Effects

Applications of deep learning to automatic multitrack mixing are largely unexplored. This is partly due to the limited available data, coupled with the fact that such data is relatively unstructured and variable. To address these challenges, we propose a domain-inspired model with a strong inductive bias for the mixing task. We achieve this with the application of pre-trained sub-networks and weight sharing, as well as with a sum/difference stereo loss function. The proposed model can be trained with a limited number of examples, is permutation invariant with respect to the input ordering, and places no limit on the number of input sources. Furthermore, it produces human-readable mixing parameters, allowing users to manually adjust or refine the generated mix. Results from a perceptual evaluation involving audio engineers indicate that our approach generates mixes that outperform baseline approaches. To the best of our knowledge, this work demonstrates the first approach in learning multitrack mixing conventions from real-world data at the waveform level, without knowledge of the underlying mixing parameters.

1. INTRODUCTION

In the post-production process, the audio engineer is tasked with creating a cohesive mixture of the recorded elements. This process involves a number of technical challenges [1], such as reducing masking, ensuring balance between the sources, and addressing noise or bleed, as well as artistic considerations, such as selecting the timbre and level of the artificial reverberation. Producing a mix is especially challenging due to the interplay between the aforementioned tasks, which are often dependant on each other, and not easily separated.