An AI system for evaluating pass fail in fundamentals of laparoscopic surgery from live video in realtime with performative feedback

Medical students preparing to be a surgeon are required to demonstrate proficiency in laparoscopic surgery as part of their training. This is done via simulation on a Fundamentals of Laparoscopic Surgery (FLS) kit where the student has to use graspers to transfer six rings from one set of pegs to another and then again back to the original set of pegs without dropping the peg. A peg dropped outside the box is considered a fail but a drop inside is the box is allowed as long as the transfer resumes with the correct grasper. We present an AI system that automatically determines if a student has passed or failed the FLS test. Our system uses an underlying YOLO model to detect the FLS box, the left and right graspers, and the FLS pegs and rings. We then use logic on top of this to detect events such as pick peg from ring, transfer peg between grapsers, and place peg on ring. We are also able to detect if the grasper drops the peg inside or outside the box (the latter being an automatic fail) and if the dropped peg was picked with the correct or wrong grasper. Our system detects these events in realtime without looking into future frames of the video - this means it can give performative feedback to the student as they are performing the task. To evaluate our system we trained it on 6 videos of junior medical residents performing FLS containing several instances of dropping pegs plus 1 video of a fake FLS showing deliberate drops across the board - this is so that the model can learn to identify drops. To evaluate our model we tested it on 14 videos on which our model correctly predicted pass fail on 11 - giving an accuracy of 78.6%. Compared to previous work our system requires only one camera, detects drops inside and outside the FLS box, produces a fully automated pass fail determination, and gives live performative feedback as the student is performing the task - thus informing them of their mistakes in realtime.