Multi-view hand tracking using epipolar geometry-based consistent labeling for an industrial application

This paper addresses a visual tracking and analysis method for automatic monitoring of an industrial manual assembly process, where each worker sequentially picks up components from different boxes during an assembling process. Automatic surveillance of assembling process would enable to reduce assembling errors by giving early warning. We propose a hand tracking and trajectory analysis method from videos captured by several uncalibrated cameras with overlapping views. The proposed method consists of three modules through single-view hand tracking, consistent labeling across views, and optimal decision from multi-view temporal dynamics. The main novelties of the paper include: (a) target model learning with multiple instances through K-means clustering applied to accommodate different levels of light reflection; (b) optimal criterion for consistent labeling of tracked hands across views, based on the symmetric epipolar distance; (c) backward correction of mis-detection by combining epipolar lines with previously tracked results; (d) a multi-view voting scheme for analyzing hand trajectory using binary hand location maps. Experiments have been conducted on videos by multiple uncalibrated cameras, where a person performs assembly operations. Test results and performance evaluation have shown the effectiveness of this method, in terms of multi-view consistent estimation of hand trajectories and accurate interpretation of component assembly actions.