Manufacturing Operator Ergonomics: A Conceptual Digital Twin Approach to Detect Biomechanical Fatigue

The primary sources of injuries in the workplace are improper manual material handling (MMH) motions, forklift collisions, slip, and fall. This research presents a Digital Twin (DT) framework to analyze fatigue in humans while performing lifting MMH activity in a laboratory environment for the purpose of reducing these types of injuries. The proposed methodology analyzes the worker’s body joints to detect biomechanical fatigue as a factor of change in back, elbow, and knee joint angles. Using the dynamic time warping (DTW) algorithm, the change in joint angles with time was analyzed. The variation in DTW parameters was evaluated using exponentially weighted moving average (EWMA) control charts for further analysis. A preliminary study considering two healthy male subjects performing seven experiments, each under an optical motion capture system was performed to test the model’s efficiency. Our contributions are twofold. First, we propose a model to detect biomechanical fatigue in the subjects performing MMH lifting activity as a change in joint angles. Secondly, the research also shows evidence that different individuals show signs of body fatigue via different body joints and showcases the need for a true personalized DT for an operator for fatigue assessment in an MMH environment.


I. INTRODUCTION
Industry 4.0 (I4.0) is the era of digitization and is evolving exponentially. Technologies such as the Internet of Things, cyber-physical systems, enterprise architecture, artificial intelligence, robotics, autonomous vehicles, and 3-D printing have been vital components in the development of I4.0 [1]. The futuristic manufacturing environment focuses on shorter delivery times, increased levels of customization, product variety, quality, and demand variability [2]. A critical player of technology in this era, which is transforming manufacturing processes, is the digital twin (DT). Although mathematical models have been used to evaluate manufacturing performance, I4.0 tools have made the interaction of physical and digital systems possible. With the transition in manufacturing to more technologically advanced and automated systems, it is essential to understand the human operators' role in these environments. I4.0 is accompanied by a change in the range of worker tasks and demands in the factory context [3]. The operators now are multi-skilled and perform jobs at several workstations capable of responding to mass-customized products and processing large amounts of information [4]. Romero et al. [5] has defined this innovative generation of operators as Operator 4.0, i.e., humans assisted by machines and technology to enhance their physical, cognitive and sensorial capabilities to perform their manufacturing tasks. A significant amount of manufacturing and material handling activities are highly manual [6]. Material handling is one of the most physically demanding tasks, and thus, can quickly become a leading factor contributing to operators' accumulation of mental and physical fatigue [6]. According to the Bureau of Labor Statistics, 114 million people were employed in the Warehousing and Storage Industry Group in 2018 [7]. These statistics show that 22% of the workforce are "Laborers & freight, stock & material movers, hand," and 17% work in stocking, order filling, or packaging by hand. According to Reeve et al. [8], the industry sectors "Construction," "Transportation & Warehousing," and "Manufacturing" reported a cumulative total of 13,782; 10,952; and 5,177 fatalities from 2003 to 2016, respectively. The study also listed that "Backincluding spinespinal cord" was the most frequently injured body part, with 17% of cases involving days away from work and about 36% of cases involving sprains and strains. The research highlighted that the human operator needs a safer environment to work. The substantial reduction of fatalities over time proves the use of safer practices and corroborates the need for continued research in this field. Factors such as personal characteristics, training, experience, and health conditions can influence fatigue in workers, impacting activity performance [6]. While performing basic manual material handling tasks, workers experience physical fatigue due to repetitive activities such as lifting/loading, leading to a high risk for low back, trunk, spine, hip, and knee injuries [9]. This research attempts to identify opportunities to track biomechanical fatigue caused by traditional repetitive Manual Material Handling (MMH) lifting activity. Using fundamentals of I4.0 and DT technologies as a foundation, this research aims to conceptualize a DT of a human operator and propose a framework that enables the understanding and analysis of body fatigue while performing material handling tasks. This research conceptualizes a methodological framework enabling the creation of a personalized DT to analyze and detect biomechanical fatigue. The framework uses a dynamic time warping (DTW) algorithm and exponentially weighted moving average (EWMA) control charts to analyze the change in the human operator's joint angles.
Prior research by [10] [11] provides a framework to conceptualize the DT of an MMH operator. The framework ( Figure 1) includes the following three modules: 1) Data Collection Module, 2) Operator Analysis & Feedback Module, 3) Digital Twin Module. The Data Collection module uses a motion capture (MoCap) system and biometric suits to record human motions and body biometrics, respectively, while performing an MMH task in a laboratory environment. The system tracks reflective marker positions attached to a human subject's body and provides accurate 3D coordinate data along with body metrics such as heart rate, breathing rate, minute ventilation, and cadence. However, the data analysis of body metrics is not part of scope of this study. The data collected are then stored in a database for further analysis. The Operator Analysis & Feedback Module conceptualizes data analysis, optimization, and forecasting based on joint movements, type of motion [11], and biometric factors [12]. The Digital Twin Module presents the visualization of analyzed operator statistics in a simulated environment as feedback to the operator and stores the operator activity metrics for further research and development. This work contributes additional capabilities to the original module design [10]. Expanded capabilities of the module include the incorporation of joint angles to measure biomechanical fatigue. Keeping the idea of creating a true DT of an operator, this study provides insights on the following research hypothesis: "Different human subjects reveal biomechanical fatigue via similar body joints." Formulating the hypothesis, a binary metric was defined for the detection of fatigue via the proposed methodology, is the subject number. The value of j ranges from 1 to 6, linked to left-back (1), right back (2), left elbow (3), right elbow (4), left knee (5), and right knee (6) joints. Null ( 0 ) hypothesis (Equation 1) tests for equality in the rate of detecting fatigue by the proposed methodology for two different individuals ( ), considering the same body joint ( ). Alternative ( 1 ) hypothesis checks for the dissimilarities. 0 ∶ 1 = 2 ; j = 1,2,3,4,5,6 (1) 1 ∶ 1 ≠ 2 ; j = 1,2,3,4,5,6 (2) This research is expected to achieve the following outcomes: • Detect biomechanical fatigue in subjects performing MMH lifting activity as a function of change in joint angles. • Search for evidence supporting that, different individuals show signs of body fatigue via different body joints. • Show some benefits of having a personalized digital twin for an operator performing MMH lifting tasks.
This article is based on thesis work by Sharotry [13]. The paper is organized as follows: Section 2 provides a detailed literature review on the use of DT technologies and the preexisting methods to analyze human biomechanical fatigue prevailing in industrial environments. Section 3 explains the developed methodology to analyze fatigue as a factor of change in joint angles. Section 4 discusses the experiments carried out to validate the proposed methodology of the DT development and the results. Subsequently, Section 5 states the summary and conclusions of this research, along with the authors' future research scope.

II. LITERATURE REVIEW
Simulation is a copy of an existing system manipulated logically to identify how it behaves in varying conditions. Whereas DT is a virtual and precise depiction of a physical system, allowing real-time analysis for detection, prediction, prevention, and optimization to increase productivity. Some practitioners have used DTs and simulation interchangeably. However, we point out the following differences between the two. Simulation provides an understanding of a physical system using numbers. It leads to time and cost advantages by helping developers to understand better real-world product behavior and elevating product lifecycle management [14,15].
While a simulation provides static information like design elements, materials, and operating conditions, starting its life as a static model, a DT simulation becomes active. Its ability to change with data flow dynamically yields more valuable information not generated by a traditional simulation [16]. A simulation model does not involve the other parts of business beyond research & development. In contrast, the continuous flow of data with a DT keeps the business provider in perfect synchronization with the business operations. To summarize, DT's use allows the developers, supply chain managers, and customers to "drive" and experience the product in real-time as it grows. Common challenges in developing a DT include data analysis, enhanced manufacturing process, and effectiveness of predictive analysis [17] [18]. The economic value of a DT varies widely based on factors like development, implementation, and maintenance costs. The abstraction level of DT ranges from the lowest component (data received & analyzed from an individual part) to asset (data from a machine, e.g., tool life for predictive maintenance), system or unit (production line in a facility), and the highest component, process (business-level view) [19]. DT technology is increasingly penetrating the manufacturing and logistics sectors. It is beneficial to the researchers and technology companies to provide a digital representation of the material handling facility and supply chain system or find the optimal conditions for enhanced performance [20]. Estimating the actual system's response to identify the factors affecting its environment and allowing communication and collaboration between other simulation models and DTs, companies such as TESLA, GE, and DHL have been working towards building DT versions for vehicles, engines, and warehouses, respectively [21] [22]. In 2017, the global DT market size was valued at USD 2.26 billion, with an estimated compound annual growth rate of 42.7% from 2021 to 2028 [23].
International Data Corporation anticipates that 70% of manufacturers will use DT technology to conduct simulations and scenario evaluations by 2022. Unilever achieved a one to three percent increase in productivity in Brazil by using DT to cut down its facility's energy use, which led to approximately $2.8 million in savings [24]. Besides using DT technology in the "Automotive & Transport" and "Manufacturing" industries, scarce research has been done to use the concepts of DT to evaluate the well-being of human operators in industry.
Since 1972, data from the "Injuries, Illnesses, and Fatalities" program of the US Bureau of Labor Statistics show that injuries and fatalities incurred by workers have decreased [8]. However, the results show that there is still a need for research to make workers safe on the job. As human involvement plays an essential role in the productivity of the system, risks related to MMH tasks associated with the nature of the load, type of task, work environment & the operator [25] allow the use of technology to analyze the ergonomics of the work environment to reduce worker fatigue [11]. Multiple tools have been introduced to measure fatigue in the workforce. Methods such as standard questionnaires after completing a job or using on-body sensors have been used in the past to analyze fatigue in construction workers [26]. Despite their use, the on-body sensors tend to cause discomfort for the human workforce while performing tasks. Various ergonomic assessment tools such as Rapid Upper Limb Assessment (RULA) [27] and the job strain index method [28] are commonly practiced in industries to identify repetitive movements. To identify strained postures, observational tools like Rapid Entire Body Assessment (REBA) [29] and the Ovako Working Assessment System (OWAS) [30] provide feedback based on an experienced user's scoring system. The National Institute for Occupational Safety and Health's (NIOSH) lifting equations [31], Snook tables [32], and Liberty Mutual tables [33] provide information on safe load capacity. The commonly used Borg scale [34] assesses fatigue by subjective worker feedback. Donisi et al. [35] performed a preliminary study to evaluate the use of machine learning (ML) models to classify the risk of fatigue accumulated by biomechanical overload according to the revised NIOSH lifting equation. The ML model classified the activity into a binary metric of risk and no-risk. Most of the methods described above require post-experiment evaluation, resulting in an inability to provide real-time feedback to the operator. Lack of including the object's weight, a significant component of MMH tasks, as a decision variable is considered another limitation of currently used methods. In recent times, virtual human factor (VHF) tools such as virtual reality [36], digital human models [11], and discrete event simulation allow users to perform an ergonomic assessment of systems not yet constructed [37]. A novel tool created by Greig et al. [38] used methods like biomechanical regression modeling and Methods-Time Measurement to predict worker demand and element-time, along with assisting the user inline layout and task balancing. However, this tool possesses the limitation of being used only in the design stage of the process. The study was also restricted to light assembly work and only considered loads at the shoulder joint. Research by Visentin et al. [6] proposed using energy expenditure to measure MMH workers' physical fatigue. The study induced a model for fatigue accumulation and rest allowance but was limited to less demanding activities. Activities where workers experience high energy expenditure rates due to repetitive movements were regarded as drawbacks of this methodology [6]. The study by Vignais et al. [39] and Boocock et al. [9] have proven real-time feedback for ergonomic evaluations to reduce the risk of musculoskeletal injuries. Using sensors, auditory and visual feedback by the operator reduces injury risk. A personalized digital athlete was built by [40] to create a digital version of on-field athletic performance. The use of big data architectures allowed the creation of a personalized "digital athlete." Based on extensive datasets, missing individual data was estimated to reduce the use of traditional experimental designs to evaluate humans' ergonomics in the sports biomechanics community. Researchers proposed using a Deep Learning Neural Network (DNN) scheme to estimate the missing data (ground reaction forces) using only MoCap trajectories as the input. A concept of human DT for fitness tracking and management was showcased by Barricelli et al. [41], monitoring a team of athletes. The DTs analyze collected data, providing suggestions to the trainers for optimization actions in athletes' behavior. Romero et al. [5] introduced the concept of Operator 4.0, i.e., humans assisted by machines and technologies to enhance their physical, cognitive, and sensorial capabilities to perform their manufacturing tasks. Chan et al. [42] provides a detailed review on the use of ML techniques to prevent work-related musculoskeletal disorders (WMSDs). Artificial neural networks (ANN) and decision tree (DT) based models were found to be the most frequently employed ML techniques. However, the need of personalized ML models was considered to further improve the accuracy of developed models. Discussions on the use of wearable tracking, Augmented Reality (AR), Virtual Reality (VR), robots, exoskeletons, and data analytics to enhance worker's capabilities were included. Jimenez [20] explains the opportunities of building industrial DTs of material handling operators. These include: 1) training based on digital copies of highly skilled operators, 2) real-time ergonomic evaluations and feedback, 3) workplace optimization and testing, 4) personalized health plans, and 5) communication between human-based and equipment-based DT agents. Hernandez et al. [43] proposed Recurrent Neural Networks (RNN) to predict the human body's motion, predicting human operators' fatigue for the specific material handling operation. Lee et al. [44] reviewed the recent developments of using machine learning techniques to address ergonomic issues in manufacturing. Since the literature shows that fatigue has a protective function against irreversible muscle damage, it plays a vital role in the redistribution and reorganization mechanisms of the human body, optimizing active muscle fibers and multi-joint coordination [45]. Al-Mulla et al. defines the biomechanical fatigue, measured considering body kinematics and kinetics, i.e., speed, velocity, and acceleration [46]. Dingwell et al. shows that a direct link exists between localized muscle fatigue and alterations in movement kinematics [47].
Objective measures for muscle fatigue include surface electromyography (sEMG) [48], wearable electromyography [49], mechanomyography (MMG), and ultrasound strain imaging [46]. A limitation of using sMEG is that it only provides a measurement for the muscles where the sensor is positioned. Postural control and movement coordination also vary with the accumulation of fatigue. With the accumulation of muscular fatigue, changes are observed in movement kinematics such as range-of-motion (ROM) and angular velocities [50]. An optical MoCap methodology with infrared cameras is preferred to measure the change of body kinematics accurately. Tracking individual joints with MoCap technology allows an in-depth analysis of the movement. As concluded by M. Golan et al. [51] and M. Peruzzini et al. [36], the lack of standard datasets makes it challenging to validate standard human behavior and the real-time mapping of operator movements in a factory environment. The literature decision matrix (Table 1) summarizes the studies discussed in the previous section. This matrix classifies tools used for MMH activity assessment on a factory floor based on the following factors: evaluation of human ergonomics (EE), feedback to the workers (WF), ability to detect upper and lower body biomechanical fatigue (UB/LB-BF), real-time capabilities or applications (RT), consideration of repetitive motions (RM) and handling of loads (HL) while performing the MMH task. The matrix represents that research by [26], [27], [28], [29], [30], [39] and [51] presents various tools for ergonomics evaluation of an operator performing an MMH task. Comparing the literature, not all the proposed methodologies focus on real-time worker feedback, fatigue analysis based on all body joints, and repetitive motions done by an operator. [26], [27], [30], and [39] hold the capabilities to provide worker feedback but only [39] tends to provide real-time feedback. Research by [9] indicated the significance of realtime biofeedback for a repetitive lifting task. The feedback enables the correction of body motion as the individual performs the task. It significantly reduces the risk of injury and educates the operator on correct methods. The weight of load also plays a critical role in MMH tasks. Experiments conducted by [31] and [32] recommend allowable loads for an MMH operator. Other included research in the matrix fails to consider the weight of the load in their respective analysis. [29], [30], [40], and [45] presented tools to evaluate all the upper and lower body joints, whereas others considered either upper body or lower body only. Current research attempts to fill the gaps in recent research to develop a personalized and real-time ergonomics assessment tool for MMH operators.
We illustrate the use of non-parametric DTW technique to evaluate biomechanical fatigue. Recently, there has been an increase in use of statistical parametric mapping (SPM) in the field of biomechanics for analysis of kinetics [53], kinematics [54] and time series data [55]. Pataky et al. [56] compared parametric and non-parametric procedures to analyze biomechanical process trajectories. The authors concluded that the choice of the procedures had negligible effects on the results [56]. However, the non-parametric algorithm such as DTW allow adequate description of a time-varying dynamic specification by revealing the dynamics between two time series and assessing the similarity [57]. DTW is also advantageous, due to its low data requirements [58], and its ability to be influenced with unseen data as the models are not based on a training dataset [59]. Current research discusses use of DTW algorithm to analyze joint angles, performing preliminary ergonomics evaluation while a human operator performs repetitive motions while handling loads. Assessing multiple body joint angles of the operator, allows the model to consider upper and lower body biomechanical fatigue. DTW's flexibility on data point mapping, allows this type of comparison by providing a trend similarity metric between two motions [60], and low data requirements [58] makes it suitable to analyze fatigue in realtime.

III. METHODOLOGY
This section discusses the methodology to conceptualize the DT framework for an MMH operator. The Data Collection module discusses the methods used to collect and measure human biomechanical data using optical MoCap technology while performing a repetitive lifting and lowering task. The collected data acts as an input to the Operator Analysis & Feedback module. This module analyzes the collected data for changes in joint angles over time using the DTW algorithm.
The results are then evaluated via EWMA control charts to detect fatigue as a change in body joint angles.

A. DATA COLLECTION MODULE
DT development features advanced big data architectures, using passive & active imaging, multi-sensor integration, data mining, real-time image processing, and non-linear data science analytics [40]. Researchers have made conclusions regarding the lack of standard datasets in material handling, VOLUME XX, 2017 6 which acts as a barrier for validating operator behavior [51] [36]. In a study by Johnson et al. [52], a significant number of data captures have been carried out for sport-specific applications, providing access to about 20,066 motion capture files. The sports-specific exercises include walking & running trials, football kicking, and baseball pitching; these exercises do not follow the motion patterns found in industrial MMH applications. Hence, the data collection module focuses on creating structural databases and training sets to develop a DT vision of an industrial MMH operator. To accomplish the vision of a true DT of an operator, it is necessary to build specific datasets with required variables. Body kinematics of the individual have been identified as essential factors to determine human skeletal muscle fatigue [50]. This novel approach collects data specific to the motions carried out by an MMH operator while performing such tasks on the shop floor. The data collection process simulates an actual MMH operation in a controlled environment, collecting the motion and biomechanical data. Previous research by Karg et al. [45] concluded that, as humans accumulate fatigue, observable changes reflect in joint kinematics such as ROM, angular velocities, and angular acceleration. Hence, optical MoCap methodology with infrared cameras is recommended to compute body kinematic parameters. MoCap technologies provide more accurate measurements than computer vision and inertial measurement units (IMU) [61].

1) DESIGN OF EXPERIMENTS
Two male subjects with similar physical characteristics and daily levels of activity were chosen for the study. Ethical approval for this study was obtained from Texas State University's Institutional Review Board (IRB: #6732). None of the participants were experienced in MMH activities. The participants performed a repetitive lifting and lowering task with 18 reflective markers on the subject's body joints based on recommendations by [62]. A fleet of twelve Qualisys MoCap cameras [63] sampling at 100 Hz tracked the reflective markers.
The subjects performed repetitive lifting and lowering of a box weighing 24 kg (52.91 lbs.) at a frequency of 9 seconds at the height of 51 cm (20.08 inches). The parameters of activity were chosen based on recommendations for acceptable weight of lift and lower by [32]. The subjects rated their physical exertion level every minute based on Borg's 20-point Rate of Perceived Exertion (RPE) [34]. The participants were allowed to continue performing the task until they sensed exhaustion or informed of the value of 18 on the Borg Scale. Subjects performed seven replicates of the activity each. Researchers ensured that the participants received rest of at least one day in between consecutive experiments. Upon completing the experiment, 3D coordinate data from the MoCap at 100Hz and RPE data at one value per minute were stored in a database for further analysis.

2) MOTION CAPTURE DATA PREPROCESSING
The MoCap data should be free of phantom markers, the marker trajectories should be free of spikes, and all the marker data should be at a fill level of 100%. The data is preprocessed manually to meet these three quality features. Due to the activity's complexity and a limited number of MoCap cameras, some of the subject markers are unidentified. Unidentified markers are a result of the MoCap system not being able to identify markers while capturing. The first step of post-processing a successful capture includes removing the phantom markers and matching these unidentified markers to their respective names. Hence, the researcher must identify such instances and identify the markers manually.
Another reason for unidentified markers is the temporary loss of markers, which does not include instances when markers fall off accidentally. Markers' temporary loss can be caused due to an obstruction between the marker and the camera pointing towards it. For example, when a subject lowers a box, and their forearm blocks a knee marker. Static, linear, polynomial, relational, virtual, and kinematic methods can fill such trajectory gaps [63].
In the MoCap process, at times, noise is received in data at unwanted frequencies. Capturing marker motions at such frequencies is irregular for humans to perform or is caused by the irregularities in the data processing by the MoCap system. The process of filtration is applied to remove such irregularities. Data filtration is used to smoothen the data received by the MoCap systems. The filtering technique allows the removal of unwanted noise and smoothening the data. Fourth-order low-pass Butterworth filtering [64] at a frequency of 6Hz was used to smoothen the data as recommended by [65]. After ensuring that the required markers for analysis are 100% filled, and there are no unwanted spikes in the trajectories, the marker 3D coordinate data can be used to measure respective joint angles. The "Analyze" tool of Qualisys Track Manager (QTM) [63] was used to measure the study's respective joint angles. The tool uses an inverse kinematics concept to measure angles from the 3D MoCap data [66]. The respective marker locations were defined in the QTM Analyze tool by selecting the markers required for measuring the six joint angles (left elbow, right elbow, left back, right back, left knee, and right knee). As a result, all the recorded frames' joint angles were calculated and exported in the database for further analysis.

2) DATA SEGMENTATION
The six joint angles and marker trajectory data for the performed experiments were merged via R [67] to proceed with segmentation. The experimental data include lifting and lowering motion performed by the subject. The scope of this research is limited to the "lifting" motion only. A segmenting filter [68] was developed in R and implemented via RStudio [69] using the plyr library [70] to trim the activity performed during the lifting task. The segmenting filter uses sliding window type of segmentation algorithm, to provide a piecewise representation of the time series data. Authors would like to refer the reader to [68] for more information on the segmenting filter.
The activity was divided based on "Motion with Load" and "Motion without Load." Motion with Load is the activity carried out by the subject as they grab the lifting container and then lift it to its location of placement at the required height. This Motion with Load is identified as a "segment." The segmenting filter identifies and splits the data for analysis into the desired lift segments. Considering a sample of raw data for illustration, Figure 2 shows the change of left elbow joint angle in degrees (y-axis) with time in centiseconds (x-axis). The plot shows change in joint angle of a subject while performing the activity, i.e., normal stance, lifting the box with weight (Motion with Load), and moving back to the original position. The lifting segment in the set of motion, i.e., Motion with Load, ranges from frame number 2749 -2885, highlighted in green. The segmenting filter is applied to the dataset, including marker trajectories and joint angles. As a result of this filter, the joint angles of segments, including only the lifting motion, are attained. Figure 3 shows the change of left elbow joint angle in degrees (y-axis) with time in centiseconds (x-axis) for multiple segments. The segmented lift motion discussed in Figure 2 is displayed as "Seg2" in Figure 3. Summarizing the concept of segmentation: a subject performs consecutive 100 repetitive lifting and lowering motions. The segmenting filter will filter out the 50 lifts and 50 lowering motions with respective marker trajectory and joint angle data associated with each frame.
Results for ten individual segmented data (Motion with Load), extracted from the original complete motion file, are displayed in Figure 3. Solid lines depict the change in angles during the first five segments (first five iterations of the activity), while subject reported RPE 6. The dotted lines show the last five segments (the last five iterations of the activity) of the recorded activity at RPE 18, reported by the subject.
Comparing the line plots in Figure 3, two observations can be made: a substantial difference in change in angle; and variability in the duration of each lift. At the beginning of the lifting experiment, the subject's absolute elbow joint angle varies between 116° and 129.3°, whereas at the end of the experiment, the same angle lies between 87.22° and 105.41° (towards 180° = elbow extension; towards 0° = elbow flexion). As a result of this change in the joint angle, it may be conjectured that in the initial five lifts (Seg1 -Seg5), the individual started with their elbows more extended and moved through a greater ROM. Towards the end of the activity (Seg65 -Seg69), the elbow extension reduces towards the beginning of "lift," but the elbow flexion remains almost similar. Overall, there is some evidence suggesting that the ROM was less when the subject was fatigued.

1) DYNAMIC TIME WARPING
Once the segmented joint angle data is in the database, the DTW method is used to analyze the change of joint angles during the activity. DTW is a technique of analyzing dissimilarities between two selected time series. It calculates the overall distance between two-time series as the series are "warped" non-linearly in a similar time domain [60]. DTW's concept emerged from the field of speech recognition in the late twentieth century [60]. Although the technique is widely used in speech recognition, its use in human kinematics is still scarce. Previous uses of DTW in analyzing body kinematics include research by Hu et al. [71] as a classification tool for basketball playing movements. Research by Ameli et al. [50] used DTW in analyzing physical fatigue in twenty subjects, performing a stair-climbing test. The joint of concern was the knee, and the study indicated positive results regarding DTW's use in determining fatigue. Multiple successful attempts have been made by researchers using the DTW algorithm in the field of activity classification, developing rehabilitation exergames [72] [73] and gait recognition [74] [50] [71]. DTW's simplicity in real-time scenarios and low requirements for training and optimizing the algorithm make it a suitable choice for the current application [58]. Hence, building the base for the use of the DTW algorithm in the field of MMH. The change of joint angles with time is a time series, and each segment is considered an individual time series for this DTW analysis. Given two time series, Query Index (Q) and Reference Index (R) (Figure 4) such that = 1 , 2 , … , , … , DTW aligns the two sequences by constructing an × matrix where the element ( ℎ , ℎ ) in the matrix contains a distance ( − ) 2 . The best match between series Q and R is the path through the matrix that results in the lowest total cumulative distance between the series. DTW is stated as: corresponds to a matrix element ( ℎ , ℎ ) which belongs to the contiguous set of elements along the path that minimizes the cumulative distance between the Query Index (3) and Reference Index (4). Figure 5 represents the alignment of two time series, outlining the process of calculating the minimum distance. Figure 5 shows the scenario when Q (solid line) and R (red dotted line) are aligned for DTW for comparison. In this research, the subject's change of joint angles in degrees with time, during the first lifting motion is considered the "Reference Index." Succeeding lifting motions are considered the "Query Index" time series. DTW distance parameters are computed for all considered joints, comparing the change in joint angles as the subject performs the experiment. For this mathematical model representation, the first segment's left knee joint angle data corresponds to the "Reference Index." The future occurring segments in the time series are selected as "Query Index." Using the first lift motion as a reference in the DTW algorithm makes the algorithm perform under an assumption. The algorithm assumes that the very first lifting motion performed is the correct motion performed by the subject. In a rare scenario, if the subject does not bend his/her knees in the first lift and performs the second lift with bent knees, the DTW algorithm will lead to a higher DTW value. The variation in knees' joint angles will be higher when both the motions are compared as a factor of change in joint angles. In the discussed example of the activity, the subject performed 215 lifts in 33 minutes (#Segments = 215), i.e., duration of the activity till the subject reaches RPE 19.   Figure 7 shows the subject's RPE scores (y-axis) at a frequency of 1 minute during the activity performed for 33 minutes, segmented into 215 iterations/segments (x-axis).
DTW distances for all the considered joints for all experiments were computed using the "dtw" library [75] available in the R software [67]. The next step of analysis includes evaluating the outcomes using exponentially weighted moving average (EWMA) control charts.

1) EWMA CONTROL CHARTS
Considering the exponentially weighted moving average (EWMA) control chart to analyze the computed DTW distances, the EWMA parameter ( ) is defined as: where, is the weight constant with 0 < ≤ 1 is the sample at = 1 denotes the first sample Starting value of 0 is the target mean. Therefore, In some cases, the average of initial data is used as the starting value.

= ̅
The upper control limit (UCL), centerline, and lower control limit (LCL) for the EWMA control chart are as follows. Parameter [1 − (1 − ) 2 ] in equations 7 and 8 tends to reach unity as its value increases. Hence, upon running for several periods, the control limits of the EWMA chart will be steady. is the width of control limits. Value of and are chosen based on the required shift in mean as a multiple of the standard deviation ( ) [76] as it leads to a shorter average run length (ARL). As recommended by [76], the fair values for and are 3 (three-limits) and 0.2 respectively, leading to the detection of a shift of three standard deviations in the mean of 2.4 ARL. In this analysis, the EWMA chart was plotted using the "qcc" library [77] available in the R statistical software [67]. Figure 8 shows a sample EWMA chart for the DTW metrics of the left knee joint. The EWMA control chart considers the weighted average of the past and current observations, making it the appropriate control chart for the computed DTW distance observations. EWMA control charts serve as an excellent alternative to Shewhart Control Charts, detecting small shifts in the process [76].
The subject's activity is divided into following three stages based on the RPE levels for the analysis: •  Figure 8 shows that the out-of-control points, denoted in red, in Stage I and Stage III points lie outside the lower and upper control limits, respectively. To enhance EWMA charts' performance, the next step of processing is estimating the control limits based on DTW values until the subject's RPE 10 rating, i.e., Stage I. EWMA control charts allow defining the upper and lower control limits based on the calibration data. The Stage I data points are taken as the "Calibration data" for the EWMA control charts. Next, the data points in Stage II and Stage III are combined and scaled as "New data" and analyzed for the EWMA control chart. Figure 9 presents the EWMA control chart used for the final assessment of change in joint angles. As the EWMA chart represents, there are 117 points above the defined control limits. This result, when aligned with the RPE values, depicts that the DTW distances tend to go out-ofcontrol (OC) when the subject accumulates fatigue (RPE > 15). The red line in Figure 9 is aligned as an indication from the subject's RPE data at RPE 15. Hence, the EWMA control chart with the defined process was selected for detecting the change in the variability of a subject's joint angles while performing the MMH lifting activity.

IV. EXPERIMENTS
In this section, the MMH experiments were conducted using the methodology presented in the "Data Collection Module" and "Operator Analysis and Feedback Module." Previous research by [78], [79] and [80] shows that in a repetitive task, with the accumulation of fatigue, different muscles cause variations in body joint angles. A significant decrease in knee and hip motion was identified by [79], and an increase in trunk flexion and knee angle was observed by [80] while analyzing subjects performing repetitive lifting tasks. The variations in joint angles can be visualized via boxplots ( Figure 10). Boxplots provide a comparative outlook on the distribution of the data. The solid box presents an upper and lower quartile range of data. The solid line in the box represents the median of the dataset. The whiskers show the minimum and maximum values except for the outliers.
Outliers are marked as values that are 1.5 interquartile ranges below or above the lower or upper quartiles, respectively, and are presented as points beyond the whiskers. Figure 10 represents the behavior of change in DTW values for six joints of Subject 1 (experiments 1 to 7). The plots are set up to compare the variability in each joint for the seven experiments.
Considering the left-back joint, the variability reduces in experiments 6 and 7. However, experiments 2 and 6 show a higher number of outliers in this joint's DTW values. The DTW points lie in the same range of 0 to 1000, but in experiments 3, 6, and 7, the boxplots show less variability. This depicts that the DTW distances lie within a shorter range, i.e., the change in left and right back angles for these runs were almost similar to the individual's first lifting motion.
Left elbow DTW distance boxplots show reduced variability in experiments 1, 5, and 7, whereas the points lie in almost similar ranges for the other four experiments. On the other hand, right elbow boxplots lie in a shorter range than left elbow points and closer to the initial motion. The points seem to lie in the same range in the right elbow throughout seven experiments, with not more than three outliers. The maximum values of the left elbow DTW parameter lie between 1500 to 2000. In contrast, the maximum limit is attained around 1500 in the right elbow, indicating that the subject's left elbow joint shows higher DTW values with the accumulation of fatigue as recorded by the change in RPE score from 6 to 18. Left knee joint DTW values range from 0 to 900, whereas the maximum values for the right knee were around 1500, implying higher deviations in the right knee joint angle. A higher number of outliers can be seen in experiment 4 in both knee joints. We infer that with higher dissimilarities in left and right knee DTW values, the subject used one knee more than the other while performing the experiments. However, the current analysis does not provide insights on using a specific joint to analyze fatigue. Summarizing experiments 1 to 7, Table 3 shows the activity statistics. Table 3 shows the number of lifts performed in each experiment, standard deviation, and mean of computed DTW distances for the consecutive joints. The last row presents the body joint with a maximum standard deviation for each run. Based on the maximum standard deviation of DTW distances for the joints and considering research by [78], [79] and [80], the right knee, left elbow, and right elbow are joints of concern for the respective individual. Left knee joint DTW parameters lie in the range close to the right knee. The higher variation in data was recorded in experiments 8 and 9 for the right knee compared to experiments 10 to 14. Table 4 presents the activity statistics for experiments 8 to 14.

V. RESULTS
The following sections discuss the results of the analysis and tend to answer the following questions: 1. Do control charts detect fatigue in an individual? 2. Can similar joints be used to detect fatigue in different individuals?

A. DO CONTROL CHARTS DETECT FATIGUE IN AN INDIVIDUAL?
To examine if EWMA control charts detected fatigue in an individual, the OC points in control charts for all experiments were analyzed. The proposed model was evaluated based on its sensitivity (recall) metric [81]. The model's sensitivity can be stated as the parameter defined to calculate the fraction of correct detections of fatigue performed by the model [81]. It is indicated as: = Mathematically, sensitivity can be expressed as: A True Positive (TP) detection is defined as the EWMA point in the control chart, which lies in RPE 15 to RPE 17 range and beyond the UCL. False Negative (FN) is the point that lies in the same range (RPE 15 to RPE 17) but below the UCL. Figure 12 demonstrates the TP (True +) and FN (False -) points in an EWMA control chart for the left-back joint in experiment 1.
The vertical line (cyan) presents the segment when the subject reaches RPE 10. The data points before this point are considered as the calibration points. EWMA control chart's mean, standard deviations, and control limits are decided based on the calibration data. The orange and red vertical lines in the control charts present the segments at which the subject declares RPE 15 and RPE 17, respectively. The points lying between these lines are of concern, as those points indicate that the subject reached a hard exertion level.
In the displayed control chart, the process goes OC at an early stage, i.e., at segment 38. Montgomery identified that a single OC point in the EWMA control chart concludes that the process is OC [76]. EWMA control charts for all the joints and experiments were plotted. A sensitivity score ranges from 0 to 1. A score of 0 denotes that there were no OC points detected by the control chart in the desired range, making TP zero. A score of 1 indicates that the model detected no FN values. Hence, the higher the sensitivity score, the better the model behaved in detecting OC points. Since a single point out of UCL can lead to an OC process in the EWMA control chart [76], a sensitivity score greater than zero implies that the proposed methodology worked in detecting points in the hard exertion range. The number of TP and FN points (Equation 9) were calculated for each EWMA control chart using the statistical software R on RStudio [67,69]. Sensitivity results for all experiments for the proposed methodology are presented in Table 5.

VOLUME XX, 2017 14
The proposed method correctly identified the OC points at instances with a sensitivity score of greater than 0 (highlighted in green  Since we have established that EWMA control charts can detect biomechanical fatigue in an individual, the following section tests the second research question.

B. CAN SIMILAR JOINTS BE USED TO DETECT FATIGUE IN DIFFERENT INDIVIDUALS?
This section tests the proposed hypothesis, i.e., different human subjects reveal biomechanical fatigue via similar body joints.
Restating the null ( 0 ) hypothesis (Equation 1): Where, is the rate of detecting fatigue by the proposed methodology. is either 1 or 2, linked to subject 1 and subject 2, respectively. The value of j ranges from 1 to 6, linked to left back (1), right back (2), left elbow (3), right elbow (4), left knee (5), and right knee (6) joints. The hypothesis test was performed to check if the rate of detection of biomechanical fatigue for the two subjects (Subject 1 = S1, Subject 2 = S2), using the proposed methodology was either due to chance or not. We have established that the model's sensitivity considers the number of OC points after RPE 15, and one OC point in the EWMA control chart can identify the OC process. Reevaluating the results in Table 5, entries with sensitivity greater than 0 were marked as true detection, i.e., 1 and 0 remained the same. Table 6 summarizes the data for hypothesis tests.    [85]. P-values from the Fisher's exact for six tests, i.e., for six joints, are presented in Table 7. Results with p-value lower than 0.00833, would suggest different rates of fatigue detection. The results show that the smallest p-value was attained for the left elbow joint. However, we were unable to reject the null hypothesis based on p-values. Therefore, no evidence of a difference in rates of detection between the two subjects was found. The analysis also concluded that the power of the test was insufficient, and a larger sample size was required to be able reach a more reliable conclusion. The results imply the need for further research.

VI. CONCLUSION
This study presented a DT framework for an MMH operator to detect fatigue via variation in absolute joint angles. Two modules of development were presented to analyze fatigue in an operator while performing a lifting task. In the Data Collection Module, a MoCap system was used to collect the subject's precise joint positional data performing an MMH task. Left and right elbows, knees, and back joints were studied for the accumulation of fatigue. The coordinate data from the MoCap system was used to measure the joint angles using inverse kinematics concepts. The Operator Analysis and Feedback Module compared and analyzed the change in joint angles for all the lifting experiment repetitions. The DTW algorithm was used to compare the change in joint angles with time. Keeping the variation in the first lifting activity as a reference, all the other lifting motions performed by the individual in an experiment were compared and analyzed. The variation in DTW parameters was evaluated via an EWMA control chart. The proposed method was analyzed to detect the OC points in the control chart as the subject reaches hard exertion of the activity on a Borg's RPE scale (RPE 15). To test the proposed methodology, two healthy male individuals performed fourteen lifting and lowering experiments (seven experiments each), lifting a box of weight 24 kg at a time interval of 9 seconds. The subjects were allowed to perform the MMH activity until they reached an RPE level of 18. The fatigue detection methodology was tested on the experiments and evaluated via model sensitivity metric. This research demonstrated that: 1. The proposed method detected biomechanical fatigue in all fourteen experiments via at least one body joint.
2. There is a need of further research to prove that different individuals indicate signs of biomechanical fatigue via different body joints. The outcomes of this pilot study justify the need for a personalized DT for an MMH operator to provide feedback on fatigue and activity statistics. Considering the preliminary nature of the study, main limitation is the limited sample sizes and the lack of subjects with other physical characteristics. Future work will benefit from considering larger sample with multiple repetitions of individuals with varying in genders, ages, etc. The proposed framework methodology tends to fulfill the four dimensions of a true DT, as concluded by [86], i.e., connectivity, analyzability, visibility, and granularity. The first component, "Connectivity," deals with the DT being a copy of the real system. In the framework, the Data Collection Module helps document the subject's actual moves in the activity area. The operator's real-time data collection on the factory floor with technologies like Inertial Motion capture Units (IMU) & 3D vision shall allow the concept's on-site implementation. The second dimension, "Analyzability," assists the DT in decision making, assisting the human operator in real-time. Evaluating the motion data as discussed in the Operator Analysis & Feedback module allows for this true DT development component. As discussed by researchers, the proposed DTW algorithm carries real-time capabilities. Additional parameter tuning, such as optimizing the "Reference Index," shall increase the model's performance in detecting shifts in joint angles. The current use case focuses on the evaluation of joint angles for fatigue evaluation. Upon further development, body fatigue predictions can be made based on motion data [43]. The application of this approach is currently limited to operator fatigue. However, the framework includes model testing and implementation capabilities to make real-time decisions for optimized scheduling, human resource allocation, and operator-specific job tasks. Various parameters used in the current methodology can be optimized to increase the model's sensitivity. The calibration point was decided in the EWMA control charts based on the subject's RPE 10 level. Since RPE is subjective, research by Sudarshan [12] conceptualizes how various physiological factors can be used to estimate RPE; hence, the calibration point. and values in EWMA control charts can be tailored to the subject's change in joint angles. The boxplots for experiments 6, 7, 13, and 14 show lower variations than their preceding experiments. Considering ability of humans to getting used to a task by repetitively performing it, the lower variability in DTW values in later experiments remains a topic for future research. The "Visibility" element of operator DT development can be accomplished via advanced tools and dedicated software development kit dashboards with the ability to be customized on-demand. The current framework withholds the capability to integrate multiple software [11] to represent the operator's futuristic avatar with a real-time diagnosis of lifting motion and advanced training of the new generation workforce. The discussed framework is a conceptual model with a high abstraction level, considering minimum details of the system about the development of a true DT. Whereas, the fatigue analysis of the operator sits at a lower abstraction level, considering the very specifics of the operator body fatigue based on body joint movements. With further incorporation of other body joints, forces, and torques acting on the body, joints can be studied for variation using the same methodology. Henceforth, fulfilling the initial requirement of the fourth dimension of DT development, i.e., "Granularity."