GPU-Based Realtime System for Cinematic Virtual Reality Production

Relatively inexpensive head mounted display (HMD) devices made possible by advances in digital imaging technology are moving the immersive virtual and augmented reality mediums into the entertainment mainstream. This departure from the traditional art of linear storytelling requires a richer production workflow and toolset capable of seamlessly integrating numerous potential narratives. An effective virtual reality (VR) workflow requires low-latency realtime synchronized capture of multiple live high-resolution video streams, realtime stitching and viewing of the captured assets, and realtime composition with computer generated elements optimally rendered for the HMD. This paper presents a realtime system for VR production that encompasses optimal high-bandwidth data transfers, graphics processor unit-based video stitching, and techniques for optimally rendering to the individual display characteristics of specific VR HMDs for on-set preview.