Enhanced Human–Robot Interface With Operator Physiological Parameters Monitoring and 3D Mixed Reality

Remote robotic interventions and maintenance tasks are frequently required in hazardous environments. Particularly, missions with a redundant mobile manipulator in the world’s most complex machine, the CERN Large Hadron Collider (LHC), are performed in a sensitive underground environment with radioactive or electromagnetic hazards, bringing further challenges in safety and reliability. The mission’s success depends on the robot’s hardware and software, and when the tasks become too unpredictable to execute autonomously, the operators need to make critical decisions. Still, in most current human-machine systems, the state of the human is neglected. In this context, a novel 3D Mixed Reality (MR) human-robot interface with the Operator Monitoring System (OMS) was developed to advance safety and task efficiency with improved spatial awareness, advanced manipulator control, and collision avoidance. However, new techniques could increase the system’s sophistication and add to the operator’s workload and stress. Therefore, for operational validation, the 3D MR interface had to be compared with an operational 2D interface, which has been used in hundreds of interventions. With the 3D MR interface, the execution of precise approach tasks was faster, with no increased workload or physiological response. The new 3D MR techniques improved the teleoperation quality and safety while maintaining similar effects on the operator. The OMS worked jointly with the interface and performed well with operators with varied teleoperation backgrounds facing a stressful real telerobotic scenario in the LHC. The paper contributes to the methodology for human-centred interface evaluation incorporating the user’s physiological state: heart rate, respiration rate and skin electrodermal activity, and combines it with the NASA TLX assessment method, questionnaires, and task execution time. It provides novel approaches to operator state identification, the GUI-OMS software architecture, and the evaluation of the 3D MR techniques. The solutions can be practically applied in mission-critical applications, such as telesurgery, space robotics, uncrewed transport vehicles and semi-autonomous machinery.


I. INTRODUCTION
In hazardous environments, as long as it is technically and economically feasible, robots can execute actions in the vicinity of danger to reduce risks for humans. However, depending on the conditions of such an environment and available resources, the autonomy level given to the robot must be decided. Suppose the task can be executed with high-enough reliability by following a set of formulated rules or using Artificial Intelligence (AI). In that case, an autonomous robot can be used without a constant connection with the operator. Such a task can be drone navigation in an open-space environment, recognition of objects, or manipulation of an assembly line robot. However, supervisory control or direct teleoperation is needed if the task is too unpredictable or critical decisions must be made based on a broad context. For example, during a robotic surgery executed by a remote doctor or a military robot control, the human operator should always be able to decide, interrupt and take control. Therefore, in a remote control system, the operators are a part of the system, and their state must be monitored. Mission success is inseparable from the human operator's performance and the efficiency of the tools [1]. From the operator's viewpoint, the factors contributing to the success can be divided into intrinsic and extrinsic, as explained in Figure 1. Consequently, the success of a telerobotic mission can be considered in three categories to take into account the full system change impacted by an intervention and its consequences: • The fulfilment of the intended goals (e.g. a remote inspection of a problem), the successful coping with potential unexpected events (e.g. obstacles, lost communication or additional tasks) that arose during the mission, and the time needed to accomplish it. The time can be measured only during the manipulation of the robot in such an intervention as surgery, or it can include the time needed for preparation, training and simulation for a space robotics mission [2]. Usually, the shorter the time, the better, or there is a time limit, such as a technical shutdown of an infrastructure where the mission is executed or when equipment is available.
• The state of the environment or the robot after completing the task. Suppose the environment's or the robot's value is high, for example, of the International Space Station [2] or the Large Hadron Collider (LHC) equipment in a particle accelerator. In that case, any unintended collision or damage to the robot or the environment must be avoided and will contribute to telerobotic mission success. For example, if the robot's state is the most important, during operations with a Mars rover, any commands and navigation instructions must be sent after careful consideration.
• The state of the human operator during or after accomplishing the task. Maintaining the operator's good mental and physical state by eliminating stress factors, decreasing workload, or recognising the physiological overload of the person is crucial for avoiding manipulation errors and accidents, especially in long-lasting missions. The teleoperation can cause eye or muscle strain, stress, and temporary or even permanent harm to the operator (such as a headache, dry eyes or eye discomfort [3], or anatomical changes due to unnatural posture over extended time [4]). So, if the operator is in a bad state or can no longer operate, it may be considered a mission failure despite achieving other mission goals [5].

A. TRANSITION FROM 2D TO 3D MIXED REALITY HUMAN-ROBOT INTERFACES
Telerobotics in the particle accelerator complexes requires particular navigation techniques, knowledge of varied electrical, radioactive, magnetic or gas risks, and safety procedures. The controlled robot, such as a mobile platform with a redundant manipulator, also increases the necessary expertise. The risk of completely losing connection with the robot is present due to scarce communication resources or radiation that affects electronics. In case of such an event, another robot needs to be used to rescue the lost one, or special procedures must be thought of in advance. The lack of a high-bandwidth communication system affects the human perception of the remote environment. All these requirements and risks in the particle accelerator's hazardous environment impact the operator's emotional state. Therefore, improvements in the interface are necessary to provide the most appropriate tool that mitigates the stress and makes the executed actions easier. The training process is no less critical [2], and the simulators [6] are very helpful in training before the actual intervention. Already 30 years ago, it was confirmed in [7] that adding virtual information in the perception channels during remote control greatly enhanced operator performance. In the cited work, virtual fixtures visually guiding the movement in specific directions or along certain shapes provided useful references that resulted in higher task efficiency. Moreover, the application of transparent Augmented Reality (AR) headmounted devices (HMDs) for procedural guidance in space operations [8] showed improved performance and decreased workload during astronauts' tasks. The study in [9] confirmed that immersive interfaces enhance telepresence, efficiency and situational awareness, especially for hyper-redundant robots, and were chosen by 94% of the research participants. The advantages of applying Augmented Reality Virtual Surrogates for aerial drone remote control were demonstrated in [10]. It eased distal operation and improved precise positioning and multitasking ability. Involving Mixed Reality in the robotic manipulator programming, as presented in [11], indicated the benefits of reducing program writing time and the number of errors due to virtual simulation in a virtual environment. Also, during a preliminary study and a 3D MR interface pilot project [12] at CERN, it was concluded that a transition from a 2D-based to 3D-based interfaces could bring multiple benefits, such as better spatial cognition, collisions avoidance or detection, motion planning, situational awareness, three-dimensional perception of the environment and spatial vision cues. Which, as a result, increased the capabilities and safety of teleoperation.

B. VITAL PARAMETERS MONITORING OF AN OPERATOR
The Autonomic Nervous System (ANS) [13] is a component of the peripheral nervous system that regulates involuntary physiologic processes, including heart rate, blood pressure, breathing, sweating, and digestion [14]. The ANS consists of two main branches: 1) Sympathetic Nervous System which activates body processes that help in stress or danger. It is responsible for your body's ''fight-or-flight'' response [15]. 2) Parasympathetic Nervous System which has the opposite effect to the sympathetic nervous system and is responsible for the ''rest-and-digest'' body processes.
Therefore, these two parts of the ANS usually operate antagonistically: the former activates body processes, while the latter deactivates or lowers them. This balance is crucial for the body's well-being. Since dynamic changes in the ANS in response to a stressor cannot be controlled, certain physiological signals such as cardiac, respiration, and electrodermal activities can be used as reliable indicators of stress [16]. These biometric signals are valuable because they cannot be consciously controlled, falsified or kept hidden by a person and can reveal information about the unconscious state. Such a state can be frustration, a human response related to anger and disappointment, defined as an emotional state of no possibility of reaching a target. Stress is one of its consequences [17]. The most commonly used physiological signals to detect stress state are cardiac activity, respiratory activity, brain activity [18], body temperature [19], sweating, eye movements, facial expressions and gestures. Previous studies about human health monitoring systems and the measurement of vital parameters have been considered for general purposes such as driving assistance and fatigue recognition [20], [21], office worker stress monitoring [22], coal mine worker safety [23], and remote videomediated assistance [24]. In human-robot interaction, human emotions, especially negative ones, influence the performance of robotic interventions. For example, frustration could impact performance quality [25] and cause a waste of time [26]. Several studies have been carried out on the monitoring of a person's vital parameters during activities involving the use of robots: • Implementation of a cooperative human-machine interaction system with the primary objective of adapting the robotic arm control strategy according to the operator's emotional state, such as stress and fatigue using cardiac and electrodermal measurements [27]; • In human-robot cooperation, the robot was expected to recognise the psychological state of the human through the analysis of heart rate variability to deduce the mental stress of the user during collaboration actions [28]; • For wearable robotic equipment, such as exoskeletons, the interaction between humans and robots is paramount. User's physiological parameters are monitored to evaluate stress, reduce it to a minimal level and improve the applicability of assistive robotic devices [29]; • Operator emotions, physiological involvement, cognitive workload and usability in robotic teleoperation were investigated to design affect-aware robotic systems capable of adequately mitigating negative emotional states of the operator [30]; • In robotic surgery, surgeon electromyographic signal from muscles contraction was analysed for the validation of a novel approach to robot-aided pedicle screw fixation that guarantees comparable efficiency in the screw placement with lower muscular fatigue and more comfortable postures for the surgeon [31];

C. VITAL PARAMETERS MONITORING IN HAZARDOUS ENVIRONMENTS AT CERN
The health monitoring of workers in standard situations and emergencies in particle accelerators and experimental areas [32] is essential for personnel safety. A prototype Wireless Personnel Safety System (WPSS) [33] was developed to detect environmental conditions and monitor workers' health by measuring parameters such as heart rate and body temperature. In the context of search and rescue robots at CERN, an ultra-wideband radar for non-contact monitoring mounted on a mobile robotic platform was implemented following the autonomous detection of victims to classify survivors according to their need for medical assistance [34]. Robotics coupled with contactless monitoring was also applied to estimate the heart rate of workers during work activities in hazardous environments [35]. In telerobotics, when operations lasted longer than expected, the attention and concentration levels significantly dropped. Therefore, an eye-tracking system has been implemented in the human-robot interface [36], which prevented dangerous collisions by constraining the robot's movements and decreasing its speed when the operator became distracted or was not observing the robot's video feedback. A pilot project studying the benefits of the transition to the 3D MR interface and preliminary operator vital parameters monitoring assessment was conducted in [12], which motivated the further work presented here. VOLUME 11, 2023 39557 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.

D. MOTIVATION AND PROBLEM FORMULATION
The CERN robots ( Figure 2) have been used for more than 10 years for remote maintenance to increase the CERN accelerator complex's maintainability and availability [37]. These operations require reliable and well-adapted human-robot interfaces. Therefore, during the research and development process of the next-generation 3D Mixed-Reality humancentred interface, such validation criteria as the operator state, task execution efficiency and telemanipulation safety requirements needed to be taken into account. The 3D Mixed Reality (MR) brought new solutions, such as planning, automatic approach, point cloud 3D environment and stereoscopic view, improving efficiency and safety. However, the control complexity may have increased the operator's workload. Therefore, an appropriate method of assessing the workload had to be used. The fulfilment of functional specifications and the ability to execute predefined tasks were essential. However, human factors must also have been studied to deliver an optimal solution. In the agile development and prototyping process, the users gave qualitative feedback that the 3D MR interface offered additional functionalities and advantages to the existing 2D interface. Still, a quantitative and detailed comparison with the previously used interface was needed. This quantitative data had to be gathered during the remote control of a real robot in nominal conditions, with environmental risks and stressors in real scenarios. In this study, the Train Inspection Monorail (TIM) platform [38] with a 9 degree of freedom (DOF) manipulator with a radioactive source in the end-effector was used to verify the correct functioning of Beam Loss Monitors (BLMs) in the LHC accelerator tunnels ( Figure 3). Previously, humans must have done these verifications manually because of the complexity of the task, resulting in the person receiving a limited radioactive dose despite applying all safety procedures and personal protective equipment. The new robotic solution greatly enhanced human safety. However, the teleoperation risks involved damaging the accelerator equipment and the robot. Also, local rescue or repair interventions in case of robot accident or failure were limited due to the radiation hazard. Furthermore, the FIGURE 3. The 9 DOF robotic manipulator is installed on the TIM that operates in the LHC. In this picture [12], the arm is in the deployed state. A radioactive source is placed in the end-effector, and the arm must reach the BLM at a specified distance to verify the correct functioning of the device.
constrained communication link with the remote robot caused control signal or feedback delays. All these factors were the source of stress and required the total concentration of the operator.
The interface evaluation had to be based on objectively measured operator vital parameters, task execution times, failure or collision potential, precision and workload assessment techniques. The stress level of the operator was measured with the heartbeat, respiration and electrodermal activity to quantify how each interface's use affected the operator's physiological state. The quantitative data had to be supported by observations and qualitative responses given by the operators. The learning curves of the interfaces had to be compared. A study was done on how previous teleoperation and gaming experiences influenced learning and task execution efficiency. It was motivated by existing research [39], [40], [41], [42] showing that video game players perform significantly better on tasks requiring visual spatial attention, multiple object tracking, rapid processing of information and imagery, spatial resolution, visuomotor coordination and speed. Response to stressful situations, such as a collision, unexpected event or prolonged stress, was observed together with the measurements of the vital parameters to assess if the system could reliably recognize these situations. If vital parameters become abnormal, the system could warn the operator about the stress risk and potentially damaging effects or automatically reduce speed or stop the robot.
The experimental work presented in this publication studies the following hypotheses in the context of human-robot interfaces for mobile telerobotics in hazardous environments, based on the experimental data gathered in the particle accelerator robotic scenario with a redundant mobile manipulator: 1) Hypothesis 1: While providing a safer supervisory control and a better perception of the redundant manipulator and the environment, the 3D MR interface does not increase the operator's assessed workload compared to the previous operational 2D interface. 2) Hypothesis 2: The use of the 3D interface does not lead to an increase in the heart rate, respiration rate and electrodermal activity compared to the 2D interface. 3) Hypothesis 3: The task execution times with the 3D interface are faster than with the 2D interface. 4) Hypothesis 4: Operators with more gaming experience execute tasks faster.
The paper demonstrates that the 2D and 3D MR human-robot interfaces can be compared through the proposed evaluation techniques, which steer future interface developments. The paper also evaluates the Operator Physiological Parameters Monitoring System via a human study of 12 participants and concludes on its usability and limitations.

E. NOVELTY AND CONTRIBUTION
This publication extends and contributes to three domains. The first is the domain of physiological telerobotic operator vital parameters measurements applied to evaluate the workload and stress estimation during a telerobotic intervention in a harsh environment. The Operator Monitoring System (OMS) is proposed to continuously measure the heartbeat, respiration and electrodermal activity and assess the operator's state (relaxed, normal, stressed). The system had to be flexible enough to integrate sensors, including non-intrusive and wireless ones. These experiments gave valuable feedback on whether the system was well adapted to a person's anatomy and the differences in the physiological signals between operators under real operating conditions. The feedback allowed to fine-tune the algorithms and thresholds used for signal processing.
Secondly, in the experimental part of this work, a comparison of the 2D interface to the 3D Mixed Reality interface was provided in terms of task execution times, learning curves, the NASA TLX workload assessment, vital parameters measurements and detailed feedback questionnaires. The results were also compared regarding intrinsic human factors, such as robot teleoperation and gaming experience.
Lastly, improvements in the evaluation methodology were proposed based on the conclusions from the experiments, measurements and used methodologies. They should best suit the telerobotic use cases in particle accelerators, hazardous environments, or in general, applied to situations where the interfaces are used for remote control of manipulators or platforms. The methodology could also be applied to operator training and progress indication not only based on execution time but also on measured safety, operator's stress and focus, and calibrated feedback questions.
The experimental work was performed in a stressful and real scenario in the LHC at CERN, the world's biggest and most complex machine, which required the coordination of multiple teams, accesses, and safety procedures. The results are a unique source of experience and ground for further human-robot interfacing advancement for robotics in harsh environments.

F. PAPER STRUCTURE
The paper is structured as follows: • Section II describes the controlled robot, the 2D and 3D MR interface functionalities comparison and the OMS.
• Section III describes the experimental setup: tasks to be done, the use of interfaces, how the vital parameters were measured, what were the questionnaires, subjects, used hardware, and data post-processing.
• Section IV presents the results of the experiments, which are further discussed in Section V.
• Sections VI and VII conclude the work, summarise the findings, and define future work regarding the methodologies of interface evaluation and the OMS. To better understand the problematics of telerobotics in particle accelerators and radioactive experimental areas, we recommend familiarisation with the CERNTAURO framework [37], the operational 2D interface description [36], and the detailed functionalities of the 3D MR interface [43].

II. SYSTEM DESCRIPTION
The system comprises four main elements that interact with each other: the robot, the interface, the operator and the OMS. The interrelations are shown in Figure 4. Section II-A details the characteristics of the robot and its mission tasks. Section II-B explains the differences between the two interfaces compared later in the study. Section II-C describes the OMS, physiological parameters, algorithms, signal processing, and its integration with the 3D MR interface. VOLUME 11, 2023 39559 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.

A. CONTROLLED ROBOT
The TIM mobile robot is used in the LHC's accelerator tunnel at CERN. The TIM comprises several principal wagons (control, drive, battery) and is responsible for carrying payload wagons, such as the robotic wagon ( Figure 3). The train is suspended on a ceiling rail and has a wagon carrying a 9 DOF arm. The manipulator's task is to approach a BLM while holding a radioactive source at a specified distance. More explanation of the scenario, robot and task can be found in [12] in Section 1.1 and Section IE of [43].

B. 2D AND 3D INTERFACES FOR THE ROBOT'S CONTROL
The mission executed by the robot can be split into two phases. In the first phase, the monorail train approaches the inspection area, and safety is assured by clearing the passage and defining maximum speeds in the sections of the tunnel. The train also has a laser scanner that can detect any object in front of the train's front or back in the rail vicinity and stop movement. No obstacles are expected on the train's route, or automated doors open when the train must pass. When the train moves, the manipulator is folded and stored inside the wagon, and the radioactive source is sheltered in a radiationblocking case. Therefore, an interface with video feedback has been sufficient for supervisory or manual control tasks. The radioactive source is extracted from the protective case for the manipulation phase and measurements. The folding and extraction are semi-automatised.
In the second phase, the manipulator is deployed in an environment that is not fully modelled, and obstacles are expected. The operator controls precisely the complex manipulator. This task requires high perceptual awareness due to the locations of the targets (the BLMs). Often, they are hidden behind other equipment or in the vicinity of fragile equipment. For these reasons, safety should be assured by additional means. In the standard 2D interface (Section II-B1), similarly to the train movement's supervision, safe intervention relies on the operator's experience and video feedback. There are no collision detection mechanisms, path planning, or obstacle recognition. These new functionalities have been developed for the 3D MR interface, described in Section II-B2.

1) THE 2D INTERFACE
In the 2D interface ( Figure 5), the manipulator can be controlled in two real-time control modes: 1) Joints velocity control. The joints are moved separately at the desired speed. 2) Inverse kinematics. The arm can be moved or rotated along or around the axes of the end-effector or environment coordinate systems. The speed of the movement can be adjusted. Several input devices, such as a joystick, keyboard, primarysecondary system or haptic controller, can be used. The feedback is composed of video camera streams, which can be displayed as multiple views on the screen. The operator can manually adjust the resolutions and frame rates of the video streams for the best compromise between necessary feedback quality and network bandwidth use and delays. Pantilt-zoom cameras can be rotated and zoomed to achieve the most convenient viewpoint, while standard cameras have fixed points of view. The 2D interface requires an operator with expertise in the robot's configuration and movement due to the manipulator's complexity and redundancy in the BLM robotic measurement project. A complete interface architecture description can be found in [36]. It also describes some other functionalities, such as autonomous behaviour scripting or a prototype of trajectory definitions.

2) THE 3D MIXED REALITY INTERFACE
The new functionalities provided by the 3D MR interface extend the previously used 2D interface manipulator control FIGURE 7. The precise spatial representation of the environment allows for calculating the distance between the end-effector and the nearest environmental point, which is an advantage in the approach task. In this figure, the video canvas can also be seen in the background, and torque information is shown as arrows next to the last joint of the arm. Here, the torque was minimal, displayed as green, but the arrow would turn orange or red during a collision.
capabilities with target-oriented and trajectory task specification, supervised position-based command, collision checks, movement preview, and a precise approach with distance measurement. A considerable portion of functionalities was developed thanks to the 3D representation of the robot and the perceived environment as a 3D point cloud. The extra functionalities are: • Planning of the movement trajectories with joint or inverse kinematics control (Sections II-D and II-E, and Figure 13 in [43]).
• Preview of the manipulator behaviour, collision avoidance in planning and detection in real-time (Section II-F and Figures 14,15,16,17, and 18 in [43]).
• Automatic approach of the manipulator to a selected location of the point cloud (Section II-G and Figure 19 in [43]). Figure 6 shows an overview of the interface during an operation in the LHC. A complete description of the architecture and functionalities in the 3D MR interface can be found in Section II of [43]. Figure 7 shows how the point cloud representation is used to approach an element of the robot's environment with a precise distance visualization. To cope with the network bandwidth limitations and high delays in the CERN facilities, especially when the voluminous point cloud is streamed, the Adaptive Communications Congestion Control was developed (Section II-I-4 of [43]).

C. THE OPERATOR MONITORING SYSTEM
The CERN Robot Operator Physiological Parameters Monitoring System, or shorter, the Operator Monitoring System (OMS), was designed to measure the physiological parameters of an operator during robotic interventions. The system has a modular architecture ( Figure 8) and allows easy integration with physiological sensors. It can work as a standalone application to produce and export raw and post-processed data recordings or connect to a client to provide a service. In our use-case of the service, the client was the MR human-robot interface communicating with the OMS. In this research, the system served the following two purposes: 1) Quantitative assessment of the operator's state during a mission by observing the vital parameters with the update rate of 5 s (which can be flexibly adjusted). 2) In the experiments performed in this study, to have a measurable indicator of stress or workload for comparing the user interfaces or assessing the workload during a task.
The OMS was a server for the MR human-robot interface client, as shown in Figure 8. The OMS was developed with Python 3.8.8, SciPy and Flask libraries, and the PLUX BioSignals API (Application Programming Interface). The source code and documentation of the OMS application can be found in [44]. On the other hand, the MR interface and its integration with the OMS were developed with C# in the Unity platform.

1) PHYSIOLOGICAL PARAMETERS ACQUISITION
The physiological signals and parameters considered in this study are related to the cardiac, respiratory and sweat glands' activities. These physiological parameters were chosen as the more substantial and accessible for inferring the operator's state of stress. The signal acquisition does not require invasive sensors disturbing telecontrol (e.g. helmets or sensory gloves) and does not significantly limit the operator or cause discomfort during prolonged use. The BioSignals PLUX toolset consists of an 8-channel central hub ( Figure 9) designed for synchronous physiological data acquisition of up to 8 analogue sensors simultaneously. It supports up to 10 hours of signal streaming with Bluetooth communication at up to 3 kHz sampling rate and 16-bit resolution for each channel. In this work, the following physiological parameters were selected and acquired for evaluation and monitoring of the operator's status: • Heart rate (HR): heartbeats per minute (bpm). HR values are extracted directly from the ECG signal analysis, which measures the electrical activation leading to heart muscle contraction. This requires three electrodes to be applied on the operator's body at chest height ( Figure 9), as the user's manual indicates. The recommended electrode placement described in the user manual was strictly followed for ECG signal acquisition in Einthoven configurations [45]. Cardiac activity was an excellent marker for detecting changes in the ANS activity [46]. HR value varies with gender, age, weight, medical conditions, medications, diet, and fitness level [47].
• Respiration rate (RR): number of breaths per minute (brpm). A complete breath combines two actions: inhalation, when the air is introduced into the lungs, and exhalation when the air leaves the lungs. The standard respiration rate for an adult at rest averages between 12 and 20 brpm. To detect respiration, the PZT sensor consists of a wearable chest belt with an integrated localized piezoelectric element that measures expansion changes caused by volume changes in the chest or abdomen during breathing cycles ( Figure 9). The elastic chest belt can be adjusted in length to fit different anatomies, body positions, and chest and abdomen circumferences according to the scientific literature [48] [49].
• Electrodermal activity (EDA): refers to the variation of the skin's electrical conductance in response to sweat secretion due to changing sympathetic nervous system activity. Several works identified a strong correlation between EDA signal and emotional arousal [50]. A typical EDA signal results from two additive processes: the skin conductance level (SCL), which fluctuates slowly and represents the tonic base level, and the skin conductance response (SCR), a fast-varying phasic component. Therefore, the phasic activity can be identified as bursts with steep inclines and declines in the continuous data stream. The EDA signal peak represents an unexpected event that happened to the monitored person, and the peak level is proportional to the event's intensity. The EDA sensor electrodes were applied on the skin between the neck and the shoulder ( Figure 9).

2) SIGNALS PROCESSING
The ECG signal tracing is characterized by the typical alternation of P wave, QRS complex, T wave and U wave.
The QRS-complex is the most significant part of the ECG (Figure 10a), representing the electrical activation of the ventricles that contract and expel blood from the heart. Characteristic patterns of the QRS-complex are evident in the ECG tracing, from which the person's heartbeat can be tracked. R-peaks related to the QRS-complex are the points of the largest amplitude in the ECG signal and represent heartbeats. The recorded ECG time series feed the function identifying local maximums in the data set. The function is customized by setting the parameters of prominence, threshold, and minimum distance between two successive peaks with a twofold purpose: the former is to avoid the false detection of noise peaks unrelated to cardiac activity and triggered by the motion of the operator; the latter filters out the other characteristic waves of the ECG signal, but identifies only the peaks related to the QRS-complex. The respiration pattern is characterized by a generic alternation of waves of varying amplitude depending on the depth of the operator's breath (Figure 10b). Deep breaths lead to a wider extension of the thoracic cavity and a higher stress on the sensitive piezoelectric part of the respiration belt.
On the contrary, short breaths result in less stress on the sensing part and lower the wave's amplitude. The respiration signal patterns can be affected by sudden movements, chestabdominal muscle contraction or sometimes by the operator's speech.
The EDA signal is filtered with a low-pass filter with a cutoff frequency of 5 Hz, to remove noise or other artefacts and then decomposed in its tonic (SCL) and phasic (SCR) components (Figure 10c). A low-pass Butterworth filter is implemented for the SCL with a cutoff frequency of 0.1 Hz. The SCR can be extracted by filtering with a high-pass Butterworth filter with a cutoff frequency of 0.1 Hz [51].
The HR, RR and EDA processing is described by Algorithms 1, 2, and 3. Each algorithm takes the analogue-todigital (ADC) measurements of electrical signals from ECG, PZT and EDA sensors, which were converted and scaled to their physical representations according to the PLUX manufacturer's guidelines. The constants in the algorithms were calibrated after extensive testing with several operators and set to values that worked for all of them. With the chosen sampling rate and constants, the maximum heart rate that can be calculated is 115 bpm. Accordingly, the RR limit is 42 brpm. The EDA events limit is 18 events per minute (epm), which is much above the physiological limit. The operators may need to operate a more physically demanding input device, which could increase their maximum heart rate above the algorithm limit. In this case, the sampling rate must be increased, or the DISTANCE parameter in the find_peaks() function be decreased. The physiological state of the operator (R) is estimated as a weighted average of these triggers ( Figure 11). The distinguished states are: relaxed (0-30%), standard (30-65%), and stressed (65-100%). The weights were selected experimentally according to the contribution of each physiological parameter to the state evaluation. The HR weight is 0.6 because it most responds to situational stress factors. The RR changes are slower and more long-term, and its indication combined with HR can further infer body unrest. Therefore, the chosen weight is 0.3. The EDA completes the state estimation with a weight of 0.1. It offers the fastest response to stress but can be easily influenced by temperature and arousal [52].

III. EXPERIMENTAL METHODS
The multilayered organisation of the experiments is presented in Figure 12. The participants remotely controlled an operational robotic manipulator (Figure 3) located in the LHC. The operation was done from the Control Centre ( Figure 13). The intervention scenario allowed testing the interface in actual conditions, characterised by collision risks, temporary network bandwidth and delays deterioration, and unexpected events (e.g. hardware or software failure, a tunnel light turning off, or other uncontrolled scenarios). The 9 DOF manipulator created further challenges for participants who had never VOLUME 11, 2023 39563 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. . According to the current values of the physiological parameters, three different operator's states are identified: relaxed, standard and stressed state. The HR and RR triggers are calculated as ramps shown in the two upper graphs (a and b) and were calibrated individually accordingly to a participants' baselines (i.e. the start of the ramp is at the baseline value and it finishes at 130% of its value). The EDA trigger ramp (graph c) was fixed to start at 3 epm and finish at 6 epm. In graph d, the operator's state is evaluated according to the equation. In the presented example, the current HR, RR, and EDA values resulted in the standard state (R=61%). operated a robot or had never operated a complex redundant manipulator. These conditions were sources of stress for participants.

A. TASKS AND INTERFACES USE
The tasks reflected real tasks that operators faced during BLM robotic measurements campaign [12]. There were two types of tasks: 1) An approach to a target with a specified distance from the end-effector tip, which holds a radioactive source, to the target (i.e. the BLM). The end-effector's orientation was not imposed because only the distance between the radioactive source to the device was essential. 2) An approach and then touch (a gentle push) of a target with the end-effector tip. The orientation of the end-effector was unrestricted. The only requirements In the case of the experiments in this publication, the robot was the TIM with a manipulator approaching the BLM for measurements. The operator wearing the OMS can be seen closer in Figure 13. were the applied force direction and stopping when the contact was detected. This task reflected the manipulation task of the end-effector holding the radioactive source when it was remotely inserted into its protective container.
The operators alternately used the 2D GUI and the 3D MR interface for both tasks in the experiments. There were 16 attempts for each participant, 8 with the 2D GUI and 8 with the 3D interface. 4 attempts were made with each interface for each type of task. Table 1 presents the alternating sequence of the attempts. Before the actual attempts, each participant received a familiarisation session on using the interfaces, with the possibility to try out functionalities later used to accomplish the tasks. During the attempts, communication with the experiment organisers was not allowed. Only in case of imminent danger could the experiment organiser react by advising to stop the movement or take over the robot's control. Such imminent dangers could be a potentially close distance from the equipment that should not be approached or a collision path that a participant overlooked. During the use of each interface, the operator was free to choose any interface functionality to accomplish the task. This practically meant that in the 2D interface, the control mode could be real-time control joint by joint or with the inverse kinematics, and the cameras' quality, FPS orientation or zoom. For the 3D interface, additionally, the planning mode could be used, with previews of movements, detection and avoidance of collisions with the environment or self-collisions, as well as control of point cloud acquisition, display of distances from the arm to the environment and directing the arm to a selected point on a point cloud with an adjusted offset.
A portable computer and two external monitors were used to display and operate the robot with the interfaces. The inputs were a keyboard and a mouse for both interfaces.

B. QUESTIONNAIRES
The information gathered from participants before the experiments were: • Experience in teleoperation (types and hours). • Experience in video gaming (types of games and hours). Based on their experience in teleoperation, the participants were categorized as beginners or experienced.
After finishing the tasks, the NASA TLX workload questionnaires were filled for 2D and 3D interfaces. Additional open questions about interfaces' use feedback, preferences or encountered problems were asked in a custom questionnaire.

C. SUBJECTS
In total, there were 12 participants in the experiments. The subjects were aged 23-34 years old. The categorization between gamers and non-gamers, or teleoperation beginners and experts, was based on the number of experience hours given by the participants before experiments. The participants were divided into groups based on their positioning with the median. In each group, there were 6 participants (i.e. 6 gamers vs 6 non-gamers, 6 beginners vs 6 experts). The declared gaming experience ranged from 150 to 21000 hours, with a median of 2000 hours. The teleoperation experience ranged from 0 hours to 3000 hours, with a median of 25 hours.
The 2D interface was used before experiments by 5 expert operators and 1 beginner operator. The 3D interface was used before experiments by 1 beginner operator and no expert operators. There were 3 expert operators with more gaming experience than the median, and 3 beginner operators had less gaming experience than the median. 4 gamers had used the 2D GUI before. None of the gamers had used the 3D GUI before. The described relations are presented in Table 2.
The study was performed with the Informed Consent of the participants. Personal and Sensitive Data were processed in the scope of scientific research in the context of CERN's specific activities (legal basis par. 28.5 and 29.5 of OC 11).

A. VITAL PHYSIOLOGICAL PARAMETERS
An example of all vital parameters recording of one participant is shown in Figure 14. It is visible that the ECG, RR and EDA values varied throughout the experiment. Most of the time, the heartbeat rate goes up during task execution or decreases or stabilises during rest times. The subject collided during the touch task 2D-4, resulting in an elevated HR value. In the middle graph, it is noticeable that the RR values oscillated between ∼8 until ∼22 during operation and rest. In the lower chart, there are EDA events which did not present significant patterns correlated with tasks execution, collision moment or OMS validation period.
The system was adapted to the physiology of each participant by: 1) Calibration procedure: measuring baselines of HR and RR in rest condition before taking control of the robot.
To present the diversity of these values, Figure 15 shows the distribution of participants' baselines.
2) The manufacturer's instructions for the respiration belt tightening were followed to obtain the best signal without discomfort. Different chest sizes were taken into account in the RR algorithm, based on the study in [48].
3) The parameters of signal processing algorithms were fine-tuned after being tested by more than 20 users. The HR, RR, and EDA variations were calculated during executions of tasks for each participant, task type, and interface type, and then compared. The variations HR, RR, and EDA were calculated individually for each participant as a difference between the mean HR, RR, and EDA values during the execution of tasks and the minimum recorded values. In Figure 16, it can be seen that there were no significant differences in the variations between the use of the 2D and 3D interfaces. The Pearson Correlation Coefficient (PCC) methodology was used to verify the linear correlation between all twelve participants' 2D and 3D data sets of HR, RR, and EDA. More description of the application of the PCC to compare such data can be found in Section II-C of [35], which explained how to translate the PCC value and p-value into correlation result. The analysis indicated their strong correlation with p-value < 0.0001, and PCC HR of 0.936, PCC RR of 0.969, and PCC EDA of 0.961. Therefore, using the 3D interface resulted in a similar physiological response to the 2D interface. The variations were also compared between beginners and experts or gamers and non-gamers, and they did not show any distinct differences.

FIGURE 14.
The graph presents recordings of HR, RR, and EDA values throughout the experiment of one participant. The horizontal timeline is divided into several phases of the experiment: learning, OMS testing as HR and RR validation, approach task and touch task repetitions. They are separated with annotated vertical lines. Each task is marked with its abbreviation on the upper graph (e.g. 2D-1, which means attempt 1 with 2D GUI). The HR and RR measurements were verified during the learning phase and are marked with red or blue backgrounds. The RR parameter was tested by having the participant hold their breath for up to 1 minute and then breathe with a specified rate for 1 minute to observe the increase and stabilization of the value. The HR parameter was verified with a reference device. The 16 task repetitions were done according to the tasks sequence in Table 1.

TABLE 2.
The table shows the number of participants in the beginner or expert group. It also depicts how many beginners and experts were familiar with each interface and had gaming experience.  Figure 19, for non-gamers in Figure 20, for beginner operators in Figure 21, for experts in Figure 22); and between gamers and non-gamers in Figure 23, experts and beginners in Figure 24; as well as combined gaming and remote control expertise in Figure 25.

gamers in
The approach task was executed significantly faster with the 3D interface than the 2D (on the last attempt 42% faster), and the learning curve was steeper ( Figure 17). The touch task was executed in the beginning 17% faster with the 2D interface, but in the end, the time was almost the same as the 3D, and the learning curve was steeper ( Figure 18). The beginner operators were 23% faster with the 3D interface ( Figure 21). The expert operators executed tasks 20% faster with the 2D interface on the first attempt, but on the last attempt, they were 14% faster with the 3D interface (Figure 22), which was due to the steeper learning curve. As expected, the expert operators executed the tasks 39% faster than beginner operators ( Figure 24). Moreover, the expert operators with more gaming experience were 61% faster than beginner operators with less gaming experience ( Figure 25).
It was observed in the group of experts that a more extensive video gaming experience resulted in faster learning of key bindings, understanding of controls and player movement, which resulted in faster goal achievement (expressed by an average time of all attempts).

C. NASA TLX QUESTIONNAIRES RESULTS
The results are split into 6 categories: 3 group users (beginner operators, experts, and all participants averaged) using the  Learning curves comparison of all participants in the approach task with the 2D and 3D interface. It shows that, with the 3D interface, the approach task was executed 25% faster on the first attempt and 42% faster on the last attempt, and the learning curve was steeper.
2D interface; and, correspondingly, the 3D interface used by these 3 user groups. In Figure 26, there are four graphs of the raw rating, weights, adjusted rating and overall rating results, which compare the workload assessment. In the NASA TLX assessment methodology, the ratings range from 0% to 100%, and weights from 0 to 1. The raw rating analysis indicated the following characteristics: • Mental Demand: No significant difference was found between 2D and 3D for all participants averaged. However, there was a substantial difference between beginners' and experts' demand values using the same interface. For 2D, experts had 39% less demand than beginners. For 3D, experts had 33% less demand than beginners.
• Physical Demand: The demand for all groups of participants was less than 6. Learning curves comparison of all participants in the touch task with the 2D and 3D interface. It shows that, with the 3D interface, the approach task was executed 17% slower on the first attempt, but the times were similar on the last attempt, and that is because the learning curve of the 3D interface was steeper.
• Temporal Demand: No significant difference between 2D and 3D was found. For experts, 3D had 16% more, while for beginners, 3D had 5% less.
• Performance: For all participants averaged, 3D had 11% more demand than 2D to achieve the performance. Also, there was a difference between experts and beginners. Beginners had 18% more demand than experts.
• Effort: No significant difference was found between 2D and 3D. However, there was a significant difference between experts and beginners. Experts had 23% Effort less than beginners.
• Frustration: There was a significant difference between beginners using 2D and other groups using 2D or 3D. The value was 92% higher than, for example, for beginners using 3D. The weights analysis demonstrated the following points: • Mental Demand: Average weights for all groups varied between 0.18 and 0.24.
• Physical Demand: For all groups, the average weights were lower than 0.01.
• Temporal Demand: Average weights for all groups varied between 0.18 and 0.23.
• Performance: Average weights for all groups varied between 0.24 and 0.30 • Effort: Average weights for all groups varied between 0.14 and 0.21. The value was much higher for beginners using 3D than for other groups using 2D or 3D.
• Frustration: Average weights for all groups varied between 0.10 and 0.17. The value was the highest for beginners using 2D. The adjusted rating analysis exhibited the following attributes: • Mental Demand: For all participants averaged, the demand was 11% lower for 3D. There was a substantial difference between beginners' and experts' demand values. For 2D, experts had 45% less demand than beginners. For 3D, experts had 48% less demand.
• Physical Demand: For all groups of participants, the demand was negligible.  Learning curves comparison of participants with gaming experience using the 2D and 3D interface. It shows that participants with gaming experience had a steeper learning curve. Initially, they executed tasks 8% slower with the 3D interface, but in the end, they were 22% faster than with the 2D interface.

FIGURE 20.
Learning curves comparison of non-gamers using the 2D and 3D interface. It shows that tasks were executed, on average, 11% faster with the 3D interface.

FIGURE 21.
Learning curves comparison of beginner operators using the 2D and 3D interface. The tasks were executed on average 23% faster with the 3D interface.
• Temporal Demand: For 3D, overall, the value was 12% higher than for 2D.
• Performance: For all participants averaged, 3D had 11% more demand than 2D to achieve the performance, which was similar between all groups.
• Effort: 3D had, on average, 21% more demand than 2D. The situation is different from raw ratings due to beginners' weight responses.
• Frustration: The most significant difference was registered between beginners using 2D and 3D. The value Learning curves comparison of expert operators using the 2D and 3D interface. On the first attempt, the tasks were executed 20% faster with the 2D interface, but because of a steeper learning curve, in the last attempt, the tasks were done 14% faster with the 3D interface. It shows that participants with gaming experience executed tasks 28% faster on the first attempt and 31% faster on the last attempt.

FIGURE 24.
Learning curves comparison of participants with and without teleoperation experience averaged for both interfaces and tasks. It shows that expert operators were, on average, 39% faster than beginner operators. However, the learning curve for beginners was steeper.
was 184% higher for 2D. There was no difference in Frustration for experts between using 2D and 3D. The overall rating analysis indicated no substantial differences between the 2D and 3D interfaces in each beginner, expert, and all averaged participants. However, the analysis showed a few interesting findings in contributing factors' 39568 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. weights and differences between beginner and expert operators. These observations were made: 1) The Mental Demand in adjusted rating was much lower (47%) for experts than for beginners. 2) Performance contributed to the workload (in terms of weight) the most (27%), then Temporal Demand (22%), Mental Demand (21%), Effort (17%) and Frustration (13%). The Physical Demand did not contribute at all to the workload assessment. 3) Frustration while using the 2D interface was much higher (184%) for beginners than other groups using 2D and 3D interfaces. 4) It must be noted that variations of answers were significant between participants for most parameters as indicated by the standard deviation bars in Figure 26, which broadly overlapped for both interfaces. However, the Pearson Correlation Coefficient (PCC) methodology was used to verify the statistical relationship between the 2D and 3D responses of all participants for each adjusted rating. The analysis indicated their correlations: • PCC Mental of 0.703 and p-value < 0.01; • PCC Physical of 1 and p-value < 0.0001; • PCC Temporal of 0.927 and p-value < 0.0001; • PCC Performance of 0.702 and p-value < 0.01; • PCC Effort of 0.683 and p-value < 0.015; • PCC Frustration of 0.778 and p-value < 0.003; All ratings showed higher moderate or strong correlations. Therefore, the ratings for an individual participant were correlated between the 2D and 3D interfaces, although there were significant inter-subject differences.

D. CUSTOM QUESTIONNAIRES
After the experiments, the participants filled in a questionnaire with several questions about the 2D and 3D interfaces and their comparisons. These detailed questions allowed to gather valuable insights into the reasons for the operator's higher workload and the interfaces' potential improvements. The listed questions with grouped detailed responses are in Appendix.
In summary, based on the questionnaires filled by participants, the following advantages were identified in the 3D interface in comparison to the 2D interface: 1) Better immersion in the scene. The 3D interface generally gave more confidence, especially to beginner operators. All participants highlighted that the 3D representation of the arm in the 3D environment was one of the best advantages of the 3D interface.
2) The point clouds complemented the video feedback information. It was confirmed that the point cloud helped approach a target accurately and avoid or detect collisions. 3) Several subtasks were more manageable in 3D than in 2D control: visualizing commands, trajectories and movement execution. The 3D feedback and advanced functionalities, such as a normal point selection, made the approach with a precise distance easier. 4) Safety was more assured by higher environmental awareness, collision detection and avoidance with the preview function. 5) The estimation of distances was easier with the 3D model, point cloud, and the numeric indicators of distances. 6) The 3D representation was fully confirmed to be beneficial for understanding the arm pose better. 7) Most participants (apart from expert operators who already operated the arm with the 2D interface) agreed that after working with the 3D interface, they better understood the joints' movement with the 2D interface. Therefore, the 3D interface could be used as a training tool before using the 2D interface, which requires more advanced knowledge. 8) If participants were asked to select the interface depending on the task, the 3D interface was preferred by all participants for the task of approaching a target behind obstacles with a higher risk of collision. For a task of simple approach, the preference was split (5 for 3D, 7 for 2D) due to not much use of advanced 3D functionalities. For the touch task, the choice was also split (7 for 3D and 5 for 2D); some participants used the advanced functionalities to detect the touch, while others preferred to depend only on the visual camera feedback. Figure 27 presents the results of the polls.
The following future improvements in the 3D interface were identified: 1) Simplification of complex controls (key bindings, input device, control modes). Using a different input device (space mouse, joystick or gamepad) for a player or arm movement. 2) Adding a possibility to control multiple joints simultaneously in joint control mode. VOLUME 11, 2023 39569 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply. The lower right chart shows the overall ratings. The scale for ratings was 0-100%, and for the weight, 0-1.

FIGURE 27.
Operators' confidence comparison and choice of interface for three tasks with the 2D and 3D interfaces. The 3D interface received more votes in terms of confidence during arm movement and for a difficult approach task.
3) Making the transition to control like with the 2D interface easier (smooth switching between 2D and 3D interface depending on the task). 4) Improving point cloud visualization with surface reconstruction. 5) The operator's attention should be drawn to the physiological state display in the Head-Up Display more when there is stress.
Regarding the OMS, in the opinion of most participants, the system correctly measured their heartbeat and respiration signals. Only sometimes, the respiration value was prone to noise from movements or speaking. The participants observed no clear correlation between EDA and stress.

V. DISCUSSION
The discussions in this section focus on NASA TLX assessment and questionnaires methodology considerations, the concluded need for metrics in the training tool for operator training, and the OMS.

A. NASA TLX ASSESSMENT AND QUESTIONNAIRES METHODOLOGY
The NASA TLX methodology showed some differences between the interfaces. However, it did not fully reflect the complexity and factors contributing to the workload in the telerobotic context and could only serve as a general indicator. The specific contributing factors in NASA TLX subscales must have been deducted from participants' additional questionnaires and their descriptive feedback during interviews after the experiments. For example, the Mental Demand in the 3D interface was decreased by a higher spatial awareness and 3D representation of the robot, which helped to understand the pose. But the workload increased by a multiplied number of keyboard bindings necessary to remember or look up.
Using the NASA TLX standard protocol gave too much room for participants' interpretation of the Effort or Performance subscales. For example, Performance was understood as the minimum time needed to finish a task, the safety of the operation, or the smoother control and sureness of all movements.
Therefore, the next attempt would be to create a more precise human-robot interface workload estimation with interface-specific calibrated questions reflecting the areas of potential improvement. The additional criteria could be focused on the interface inputs' workload (to check if using a joystick, space mouse, gestures, or hand tracking is easier for a particular task). In a hazardous environment, safety is a higher priority than the time of execution. Therefore, an estimation of how much safety assurance and what usability tools (such as collision avoidance or detection) the interface provides is needed. The operator's reactions and an interface's adaptation to unexpected situations should be measured. In limited network scenarios, communication and feedback delays are a source of frustration and stress. Hence, an estimator of how the interface was coping with such a problem is crucial. For example, the interface could automatically adjust speed to the delay or modify feedback quality to minimise the delay and avoid the network collapse; or have more supervisory control, such as position-based trajectories, which are more resistant to unstable communication. All these techniques can minimise the operator's workload by explicitly addressing the delay problem that contributed to Mental and Temporal Demand, Effort and Frustration in the NASA TLX assessment.

B. OPERATOR TRAINING AND NEED FOR METRICS IN THE TRAINING TOOL
A person must undergo training before becoming an independent robot operator. During this process, usually, there is an expert teaching and giving feedback on the trainee's actions. However, when the process is long and complex, additional training tools such as training simulators are needed for skill perfection and for trying out different control techniques or scenarios by the newly trained operator.
In the performed experiments, some operators reacted in a time-based competitive manner, which could have provoked collisions at high velocities, or the manipulator was moved in close vicinity of the environment instead of keeping a safe distance and making the approach in the last phase. Hence, in the training tool, there is the need to create metrics that give feedback on how safe the teleoperation was, taking into account the distances to collisions, near-misses, or safe adjustment of speed in the vicinity of objects. These metrics can be easily calculated in simulation with a virtual robot and objects. When a real robot is used for training, the robot's modules' distances to the captured point cloud environment can be calculated. The number of keyboard key presses or mouse clicks could be counted for the performance quantification of the input systems. It was noticed in the experiments that the number of presses on a joint movement key was higher for beginner operators. If the feedback delay became high, operators changed to the control strategy of short pressing and waiting for video feedback to check the effect. This significantly increased the task time. There are also other methods for performance calculation in humancomputer interaction. For example, Fitt's law was used to assess the operator performance using virtual fixtures [53], or to analyse perceptual-motor performance within teleoperated or telepresence systems [54], [55].

C. THE OPERATOR MONITORING SYSTEM
The considerations regarding the OMS design, its future advancements and improvements were drawn from the practical use during experiments by multiple participants and are the following: 1) The ECG acquisition provided a good signal/noise ratio, and the R-peaks had large amplitudes compared to the surrounding data points. The ECG acquisition was reliable when the operator focused on the manipulation.
2) The RR measurements provided good feedback when the person had a stable breathing pattern, for example, when focused on the task. However, speaking or movements could be a source of chest movements that interfered with breaths. Therefore this parameter should not be used when the operator must use speech or move during the remote control. Or a specific algorithm recognising these interferences is needed to disable the measurement. Also, the physiological differences between operators required some tuning of thresholds when a breath is recognised and adjusting this parameter to each person or finding a typical value for a group of operators in the team. 3) The EDA measurement was the most experimental, but its feedback fulfilled its role in most cases, showing more activity when the task became more intense. However, the skin activity varied among participants, and threshold tuning was necessary and difficult. Similarly to other signals, it was susceptible to movements and friction between the electrodes or wires and clothing. 4) It was observed that sometimes the OMS's fuzzy logic calculated state did not precisely reflect the operator's feeling. It may have been caused by signal noise caused by movement, changing a position on a chair, contact of electrodes or cables with clothes, or simply talking. 5) Measurements of the baseline of the heartbeat, respiration, and EDA signals varied depending on the person's recent activity. Therefore, this measurement must be standardised, with a longer resting period, after a fixedtime low-activity work at a computer without stressors. 6) All used sensors require an attachment of wired electrodes to the body or wearing a respiration belt, which can generally interfere with tasks such as driving, walking or gestures. During the experiments, a few participants gave feedback that although the system did not deteriorate the robot's remote control capabilities, it was not entirely comfortable to wear due to wires. 7) During the experiments, there were no simulated stressful situations (such as increased delay, robot not responding, disconnection, alert message, turning off the light, or collision sound) due to the complexity and dangers while using an operational robot in the LHC. However, for future system testing, these simulations would be an indispensable method for validation, especially for the EDA measurements.

VI. CONCLUSION
The telerobotic tasks in hazardous, radioactive, underground and semi-structured environments are a source of increased operator stress and workload compared to non-hazardous environments. Therefore, the design, development and evaluation of adequate human-centred robot interfaces play an essential role in the success of reliable and safe missions. In this publication, two contributions aimed to improve this evaluation process and steer the subsequent developments of human-robot interfaces: the Operator Monitoring System (OMS) and the assessment methodology. Currently, in most human-machine systems, the state of the human is neglected (e.g., by using only a dead man's switch to check if the robot's or train's operator is conscious). The proposed approach of the OMS can significantly improve human inclusion in the system. The OMS and its experimental findings can be practically applied in any mission-critical human-robot interface, in which the operator's state should be monitored and the control adapted to the attention and stress. For example, it could be applied to surgical robots operated by doctors, or unmanned public and commercial transport vehicles controlled by remote operators (metro, drones, aeroplanes, semi-autonomous cars or machinery).
Often, developments assume that new or changed functionalities in an interface will improve the user experience (UX). However, a well-prepared iterative investigation with proper methods can steer this development better. The showcased customised methodology of assessing and comparing human-robot interfaces can be used directly or inspire applications where one interface must be evaluated or compared with a previous or a new novel interface.
The experimental application of the methodology and the OMS was done with 12 participants operating a real robot with a 9 DOF redundant manipulator in the scenario of the world's biggest and most complex machine, the LHC. The 3D MR interface was compared with the 2D interface, which has been widely used during the last 9 years in more than 160 real interventions, 500 performed tasks and 500 hours of operation. The results showed that the operations with the 3D MR on screens increased the safety and performance indicators while maintaining a similar workload and physiological response of operators. The experiments also studied intrinsic human factors, such as experience in remotely controlling robots or gaming, and the results can be used in operator selection and training processes.
Hypothesis 1 was confirmed. Improved understanding of the robotic manipulator thanks to the 3D representation and better perception of the environment with spatial point clouds were confirmed by detailed feedback from the operators. The NASA TLX assessment method did not reveal significant differences in the assessed workload comparison between interfaces.
Hypothesis 2 was confirmed. The 3D interface resulted in a similar physiological response to the 2D interface. Therefore, it can be concluded that the 3D interface did not cause higher physiological demand for operators.
Hypothesis 3 was partially confirmed. With the MR 3D interface, the execution of the approach task and learning process were 25-42% faster, especially for beginner operators. However, the touch task required more time for expert operators with the 3D interface, although the time difference became small (3%) in the last attempt. The difference in the previous familiarity with the 2D interface of 5 participants compared to only 1 participant who operated the 3D interface before may have been the main factor in favour of the 2D interface.
Hypothesis 4 was confirmed. Operators with more gaming experience executed tasks around 30% faster. Moreover, they had steeper learning curves with the 3D interface than with the 2D.

VII. FUTURE WORK
Future work regarding the methodologies of interface evaluation and metrics: • For interface comparisons and human-robot interface workload assessment, a specific assessment methodology must be designed with interface-specific calibrated questions for hazardous scenarios and remotely operated robots, which would assess the workload due to more specific sources, such as inputs, safety assurance, network delays and uncertainty of robot's status.
• In the training process without an expert trainer or during the perfecting phase, there is a need for a tool that can give feedback and metrics on teleoperation safety. The boundaries of safe speeds and control techniques depending on situation should be also communicated by the system to the trainee.
• In the used setup for the experiments, the operator's eyes' focus was not used. However, it can be precisely measured with head-mount devices (HMDs) such as those for Virtual Reality (VR) or Augmented Reality (AR) stereoscopic displays. This information can be used to check if the operator is currently looking at the operated robot and the scene and adjust the behaviour of the interface. This will be implemented in the MR human-robot interface that uses the AR HMD, which has been recently commissioned at CERN [56], [57]. Future work regarding the Operator Monitoring System: • Machine learning techniques and Artificial Intelligence (AI) could be applied to better recognize nonstandard situations, which are not easily filtered by analytic signal processing. Also, stress can manifest itself in different patterns under different circumstances and external conditions, which should be deeply studied. Further study must focus on a detailed estimation and quantification of particular sensors' stress recognition trust levels and testing algorithms with more participants.
• Initial tests of the OMS have been performed with the interfaces in spatial VR and AR human-robot interfaces using HMDs, where the operators must walk and use hand gestures or voice control. In such applications, the wired connections are troublesome and cause signal noise. Therefore a non-invasive heartbeat and respiration measurement should be used. Standalone contactless monitoring using radar and cardiac activity estimation with a pan-tilt-zoom camera has already been developed and successfully tested at CERN, as presented in [34] and [35], and must be integrated with the OMS.
• Other signals estimating stress and workload, such as eye tracking or electrical brain activity with electroencephalography (EEG), will be tested.

1) Which tasks or actions were easier with the 2D interface compared to the 3D interface?
• The 2D interface allowed the movement of multiple joints simultaneously in the real-time direct joint by joint control mode. Therefore, in this mode, it was faster to move the arm by usually combining two joints simultaneously, especially for experts and for repetitive tasks or simple trajectories that needed only a few movements with joints in a known environment.
• In the 2D interface, the video camera canvases were always visible, while in 3D, it was sometimes necessary to change the player's position to see the video canvas better. Therefore, the final part of the touch task, which had to be based more on video camera feedback and multiple movements with small increments, was faster with the 2D interface. In this task phase, there was less risk of collisions with the other arm elements, so the 3D collision avoidance was not much in use.
• The 2D interface was more intuitive because it had simpler controls. On the other hand, it lacked the advanced functionalities of the 3D interface. 2) Which tasks or actions were easier with the 3D interface compared to the 2D interface?
• For beginner operators, the 3D interface allowed straightforward learning of each joint movement and interpretation of the arm's pose. With the 3D representation of the arm, there was no need to look at the video camera to understand the arm's pose and prevent self-collisions.
• The estimation of distances and orientation of the end-effector was easier thanks to the additional information based on the point cloud. It benefited the approach task to keep a specific distance from the target, and the control was more precise.
• It was easier to visualize commands with a preview showing the trajectory, planning functions, and position-based control functionalities. The 3D interface allowed to ''jump'' to the target location with the point cloud normal point functionality and the planning arm. It did not require going joint by joint. With the preview, safety was assured, and then the arm could be moved directly to the target with one button and under supervision.
• It was manageable to avoid a collision. The preview functionality increased trust. Collision detection helped to know when the arm was in contact with itself or the target, especially when controlling the arm in real-time.
• The 3D interface gave better immersion in the entire space and the definition of obstacles. The combination of point clouds and video streams allowed a complete view of the area. The operator could better feel where the robot was in the environment (better prioperception). With the 2D interface, operators spent more time planning and checking movements with a restricted view. 3) 2D interface: What problems did you encounter?
• The only feedback available was the video stream, and when the communication delays were high, it was difficult to operate with significantly delayed video feedback. With high delays, the control felt unresponsive.
• Due to a camera perspective, estimating the distance to the target was difficult. Indirect visual clues, such as shadows in the environment, must have been used.
• The joint and Cartesian movement had no indicators of joint direction. Initially, the trial and error method had to be used to understand the movement directions.
• For beginner operators, no understanding of the arm's pose and environment and how each joint moves was a source of stress to use the system safely. There was always a collision possibility, which could not have been noticed. 4) 3D interface: What problems did you encounter?
• The 3D interface had advanced functionalities, but the input system and key bindings were difficult to remember, although there was a screen to look them up. It had too many modes of control. However, closer to the end of the experiment (after ∼2 hours of practice), most participants remembered the key bindings fluently without external help. VOLUME 11, 2023 • In the beginning, the locomotion around the scene was not intuitive for operators without gaming experience (the QWEASD keys, SHIFT to accelerate, and mouse to look around or interact). Therefore, there was an additional workload of moving the viewer (player) in the scene in addition to controlling the robot. Another input device, such as a space mouse used often in the 3D design software, could be better for player movement.
• With multiple functionalities used simultaneously (normal point, planning arm, trajectories, FABRIK end-effector control, collisions, torque arrows), the display became too cluttered, and it wasn't easy to see the environment.
• Representation of the point cloud as points but not surfaces was not natural, although it was possible to see the shapes. Sometimes, using the normal point required much time to define the target. A more intuitive method of changing its position and orientation would be better.
• The planning with the inverse kinematics and FAB-RIK mode allowed for swift deployment, but an inappropriate use could place the arm in a strange configuration. It required more practice before it could be used efficiently.
• For a few participants, the use of the point cloud for the touch determination was not always reliable due to point cloud resolution (for example, 10 mm) and system mechanical elasticity -the wagon with the arm could tilt, and the previous point cloud reading became shifted in comparison to the actual position. The video feedback for that task was more reliable in seeing the arm-environment contact. 5) Did the vital parameters displayed in the interface correspond to your stress, workload or feelings?
• Most participants answered: ''The heartbeat matched the real feeling. It corresponded to my attempts to improve the time and performance.'' • ''The respiration was usually correct. But when I was moving, it was overestimated.'' • Four participants did not pay much attention to the indicators. They were focused more on the tasks.
• Participants could not tell if the EDA measurement was correct. 6) If you could choose which interface, 2D or 3D, for different tasks, which one would you choose? With which interface were you more confident about the movement of the arm? The 3D interface received higher confidence primarily due to positioning with the 3D model showing the arm position, especially when the network delay was longer.
Operators who voted for more confidence with the 2D interface explained that it was more straightforward with less complex control modes or they had already been familiar with it. The 3D interface was selected by everybody for a difficult approach task, while 2D and 3D received a similar number of votes for a simple approach or the touch task. 7) After operating with the 3D interface, did you understand the movement better in the 2D interface? Most participants (7) answered ''definitely yes'', apart from the operators who already knew and operated the arm with the 2D interface (5 participants).