New real-time methods for operator situational awareness retrieval and higher process safety in the control room

Objective: To evaluate the application of a Deep Learning based emotion recognition system for detecting operator stress, where operator stress is a proxy for Situation Awareness (SA) changes during abnormal/contingency situation management and decision making. Background: When operators are overwhelmed by stress, their perceptions, thinking, and judgments are impaired, increasing the chance of misinterpretation of events and increasing the potential for human error. The "intelligent control room" has been proposed as a possible solution for helping operators to deal with such stress. The control room comprises a variety of components used to monitor the operator and infrastructure under his/her control in an effort to optimize the performance of the human-technical system as a whole. A critical component of this control room solution is the provision of human monitoring and assessment data in order to determine the operator's situation awareness. Methods: An emotion recognition system is designed based on two Deep Learning models, the Bidirectional Long Short Term Memory network (BiD-LSTM) and the Deep Convolutional Neural Network (DCNN), in order to process audio and facial data respectively. The system is first validated against a standard corpus of expert-coded emotion data. Post-validation, a dataset of expert-coded user stress data is coded by the system for emotional valence, and these system-generated emotional readings are compared to the expert-coded stress markers to determine any significant correlations. Contribution: This research contributes to developing the idea of intelligent and automated decision-making support in situational awareness measurement systems. Such systems support users by real-time collecting and processing data, and assist decision-making based on operator behavioral patterns.