Characterizing Animal Behavior through Audio and Video Signal Processing

This article presents two instances in which multimedia systems and processing have elucidated animal behavior and have been central in developing quantitative descriptions. These examples demonstrate multimedia systems' utility and necessity in developing a complete phenotypic description. We hope that this article will spur interest in this subject in the multimedia community, so more advanced processing techniques will enter the field of quantitative neuroethology. You might have noticed that in our two examples, there was nothing very multimodal about the media techniques used. Both of these systems are transparently unimodal. This speaks to the limited crossover between the multimedia community and the behavioral neuroscientists (or neuroethologists). These examples did show, however, that the neuroscientific community can benefit greatly from incorporating multimedia techniques into their experiments and data analysis. As the walls between these disciplines begin to fall, experimental setups that are truly multimedia will likely appear. Such systems will allow complete phenotypic descriptions of animals in ethologically relevant settings, along with methods for analyzing, manipulating, annotating, and storing the resulting data. Combining these phenotypic descriptions with the corresponding genetic and neural network properties will facilitate the connection of these organization levels and lead to a more thorough understanding of brain functioning.