Using random forests with meta frame and meta features to enable overlapped audio content indexing and segmentation

Semantic data can be extracted from soundtracks to enable automated audio content analysis and effective metadata generation, which enables scene analysis, indexing and search. Existing methods classify audio at a high level into speech, music or event sound segments exclusively, discarding content information in overlapped parts. Taking into account the fact that many real soundtracks include overlapped types of audio components, nonexclusive segmentation and indexing are essential pre-processors for reliable and non-lossy audio information mining. This paper argues the importance of nonexclusive segmentation, identifies the challenges imposed by overlapped classes, and proposes the use of meta features and random forests to detect and segment overlapped audio. Proposed algorithms are presented and validation results discussed. Estimated statistical distributions about zero crossing, RMS values, discrete cosine transform (DCT) coefficients and entropies calculated from consecutive 20 ms analytical frames over a 1 second period seem to be adequate for suitable classifiers to discriminate and identify the content in question. DCT spectra of entropy further improve the performance but only slightly.