Enhancing Video Experiences for DHH Individuals through Sound-Inspired Motion Caption-based Spatiotemporal Tacton

When deaf and hard of hearing (DHH) individuals watch videos, captions are essential for them to understand the linguistic content. Current captions, however, are not suitable for conveying non-verbal sound information, such as background music, sound effects, or speech nuances. In this paper, we designed a multimodal system, Motion Caption Haptic System (MCHS), that enables DHH individuals to encounter sounds in videos through animated caption and spatiotemporal vibration patterns, supporting a more vivid and immersive experience. We elaborately designed motion captions and spatiotemporal haptic patterns for representative sound effects and spoken emotions to work well together through surveys from 27 DHH and 64 hearing participants. An evaluation with 19 DHH individuals demonstrated the capabilities and potential of the MCHS to improve their video viewing experience, along with a discussion of important issues that need to be addressed when designing multimodal captioning systems for the DHH viewers.