Visual Effect System for Projection Mapping with Japanese Drum Performance

Recently, there have been several collaborations between new projection mapping projects and various performances. To remake it, we collaborated with a traditional Japanese drum performance. Since unity is an essential factor for a drum performance, we developed two systems to increase the sense of unity between a drum performance and visual effects on projection mapping. One of the systems employed an acceleration sensor and classified different drumming patterns using deep learning. The system synchronized each drumming pattern with a corresponding visual effect. The other system was used for floor projection visual effects. In our collaborated performance, we were unable to install any sensors due to the restrictions of the event. Thus, we used an optical see-through device to operate some visual effects interactively. This system enabled an operator to place and relocate each visual effect according to the location of each performer. These two systems will help in improving the sense of unity in a collaborated performance.