A Learning-Based Approach to Parametric Rotoscoping of Multi-Shape Systems

Rotoscoping of facial features is often an integral part of Visual Effects post-production, where the parametric contours created by artists need to be highly detailed, consist of multiple interacting components, and involve significant manual supervision. Yet those assets are usually dis-carded after compositing and hardly reused. In this paper, we present the first methodology to learn from these assets. With only a few manually rotoscoped shots, we identify and extract semantically consistent and task specific landmark points and re-vectorize the roto shapes based on these land-marks. We then train two separate models – one to predict landmarks based on a rough crop of the face region, and the other to predict the roto shapes using only the inferred landmarks from the first model. In preliminary production testing, 26% of shots rotoscoped using our tool were able to be used with no adjustment, and another 47% were able to be used with minor adjustments. This represents a significant time savings for the studio, as artists are able to rotoscope almost 73% of their shots with no manual roto-scoping and some spline adjustment. This paper presents a novel application of machine learning to professional interactive rotoscoping, a methodology to convert unstructured roto shapes into a self-annotated, trainable dataset that can be harnessed to make accurate predictions on future shots of a similar object, and a limited dataset of rotoscoped multi-shape fine feature systems from a real film production.