Robust Physician Gaze Prediction Using a Deep Learning Approach

The patient-physician relationship is an integral part of primary care visits. To build a better relationship, understanding the communication between patient and physician is the key. This study focused on analyzing the gaze, one of the most important non-verbal behaviors found to influence patient outcomes. Gaze analysis often needs a manual rating process which might be time-consuming, costly, and unreliable. This research aimed to support automated analysis of physician-patient interaction using a deep convolutional neural network with transfer learning to a build robust model for physician gaze prediction. Utilizing only 3 minutes of 15 videos capturing 3 physicians interacting with different patients in a clinical setting, the model achieved over 98% accuracy for train, test, and validation sets. By visualizing the convolutional layers and comparing sample frames from different interactions, results highlighted several patterns shared across frames predicted correctly from both seen and unseen video sequences. The proposed work has the potential to informed the future design of technologies used to capture the clinical interaction and provide real-time feedback for physicians, which will contribute to the improvement of care quality.