Use of a Multiscale Vision Transformer to Predict Nursing Activities Score From Low-Resolution Thermal Videos in an Intensive Care Unit

Excessive caregiver workload in hospital nurses has been implicated in poorer patient care and increased worker burnout. Measurement of this workload in the intensive care unit (ICU) is often done using the nursing activities score (NAS), but this is usually recorded manually and sporadically. Previous work has made use of ambient intelligence by using computer vision to passively derive caregiver–patient interaction times to monitor staff workload. In this letter, we propose using a multiscale vision transformer (MViT) to passively predict the NAS from low-resolution thermal videos recorded in an ICU. 458 videos were obtained from an ICU in Melbourne, Australia, and used to train an MViT v2 (MViTv2) model using an indirect prediction and a direct prediction method. The indirect method predicted one of eight potentially identifiable NAS activities from the video before inferring the NAS. The direct method predicted the NAS score immediately from the video. The indirect method yielded an average fivefold accuracy of 57.21%, an area under the receiver operating characteristic curve of 0.865, an F1 score of 0.570, and a mean squared error (MSE) of 28.16. The direct method yielded an MSE of 18.16. We also showed that the MViTv2 outperforms similar models, such as R(2 + 1)D and ResNet50-LSTM, under identical settings. This study shows the feasibility of using a MViTv2 to passively predict the NAS in an ICU and monitor staff workload automatically. Our abovementioned results also show an increased accuracy in predicting NAS directly versus predicting NAS indirectly. We hope that our study can provide a direction for future work and further improve the accuracy of passive NAS monitoring.