A Real-Time Framework for Video Time and Pitch Scale Modification

A framework is presented which addresses the issues related to the real-time implementation of synchronized video and audio time-scale and pitch-scale modification algorithms. It allows for seamless real-time transition between continually varying, independent time-scale and pitch-scale parameters arising as a result of manual or automatic intervention. We illuminate the problems which arise in a real-time context as well as provide novel solutions to prevent artifacts, minimize latency, and improve synchronization. The time and pitch scaling approach is based on a modified phase vocoder with optional phase locking and an integrated transient detector which enables high-quality transient preservation in real-time. A novel method for audio/visual synchronization was implemented in order to ensure no perceptible latency between audio and video while real-time time scaling and pitch shifting is applied. Evaluation results are reported which demonstrate both high audio quality and minimal synchronization error.