MT-ViT: Multi-Task Video Transformer for Surgical Skill Assessment from Streaming Long-term Video

Surgical skill assessment is a challenging task for evaluating the proficiency of a surgeon with long-duration surgical video as input. Prior works mostly utilize motion, field clearness, and fine events as guidance to better distinguish different levels of skills. However, such assisted information is hard to acquire for intra-operation and leads to the limitation in processing streaming long-term video. To address the problem, we propose a multi-task video transformer (MT-ViT) to tackle the surgical skill assessment, leveraging phase recognition as an auxiliary task to provide context awareness. We adopt two task-specific tokens correlated with temporal attention and spatial attention to disentangle the multi-task training, combined with a space-time token reorganization mechanism and attention consensus loss to improve efficiency and accuracy. We evaluate on JIGSAW dataset and two long-duration surgical datasets (Cholec80, AutoLaparo) to extensively validate our method, demonstrating superior performance on skill assessment task.