Achieving Sub-Pixel Platform Accuracy With Pan-Tilt-Zoom Cameras in Uncertain Times

In this paper, we present a novel method for selfcalibrating a pan-tilt-zoom (PTZ) camera system model, specifically suited for long-range multi-target tracking with maneuvering low-cost PTZ cameras. Traditionally, such camera systems cannot provide accurate mappings from pixels to directions in the platform frame due to imprecise pan/tilt measurements or lacking synchronization between the pan/tilt unit and the video stream. Using a direction-only bundle adjustment (BA) incorporating pan/tilt measurements, we calibrate camera intrinsics, rolling shutter (RS) characteristics and pan/tilt mechanics, and obtain clock synchronization between the video stream and pan/tilt telemetry. We call the resulting method PTCEE (pan/tilt camera extrinsic and intrinsic estimation). In a thorough simulation study, we show that the proposed estimation scheme identifies model parameters with sub-pixel precision across a wide range of camera setups. Leveraging the map of landmarks from the BA, we propose a method for estimating camera orientation in real-time, and demonstrate pixel-level mapping precision on realworld data. Through the proposed calibration and orientation schemes, PTCEE enables high-precision target tracking during camera maneuvers in many low-cost systems, which was previously reserved for high-end systems with specialised hardware. Code is available at github.com/ffi-no/Paper-ptz-subpix-accuracy.