Automatic Hajj and Umrah Ritual Detection Using IMU Sensors

A smartphone can provide a wide range of practical applications and services thanks to its advanced sensing capabilities. However, the sector of Hajj and Umrah, which are rituals performed by millions of pilgrims, still lacks intelligent solutions that can improve the pilgrim experience using these sophisticated capabilities. This research aims to bridge this gap by introducing a solution that applies a real-time monitoring process to different Umrah activities (i.e., Tawaf and Sa’i) using smartphone sensors. In the proposed solution, the smartphone first tracks the pilgrim’s path with the help of inertial sensors, commonly known as the inertial measurement unit (IMU). Then, an algorithm is developed to detect and process the different activities performed by the user and provide helpful instructions accordingly for a comfortable and successful experience. The proposed system was tested and validated using real data for Tawaf and Sa’i activities. The extracted paths were compared with the GPS data for validation. Results showed that the paths were extracted effectively and the algorithm monitored both Tawaf and Sa’i successfully. The deviation between the real path and the extracted path using the proposed algorithm can be enhanced with proper GPS assessment and step-length calibration.

and PDR. SINS is specifically used when the IMU unit is 105 rigidly attached to the tracked object [23] or, in the case of 106 pedestrian tracking, mounted to the user's feet [20]. PDR, 107 on the other hand, does not suffer from this limitation [20]. 108 Considering its capability of tracking pedestrians using IMU 109 units embedded in smartphones, we choose PDR for path 110 tracking.

111
PDR technique [24], [25], [26], [27] used for path tracking 112 in this work consists of two main stages. The first stage is step 113 counting and travel distance estimation. The second stage is 114 heading estimation. Algorithms that use smartphone sensor 115 data to achieve the objectives of each stage can be found either 116 separately or combined in the literature.
Step counting is a 117 topic of interest for many researchers. Brajdic and Harle [28] 118 evaluated different step counting and walk detection algo-119 rithms at six different phone positions. The authors reported 120 that windowed peak detection methods for accelerometer data 121 are favored for step detection with an error rate of less than 122 3%. A more recent work [29] proposed an algorithm that 123 defines three measures for analyzing the accelerometer data: 124 periodicity, similarity, and continuity. A step is verified when 125 these three measures meet certain conditions. The algorithm 126 outperformed the conventional peak detection-based method. 127 Relying on measures such as similarity and periodicity may 128 require the studied walk to be uniform to a certain degree. 129 During Umrah, though, pilgrims' movements are expected 130 to be more complex considering the usual crowd situations 131 in Tawaf and Sai areas. Rodríguez et al. [30] presented a 132 model where peak-valley detection is used for step counting. 133 They used a combination of support vector machines and a 134 standard deviation-based classifier for validating the results 135 of the standard step counting strategy. Pham et al. [31] pro-136 posed an algorithm that finds suitable minimal peak detec-137 tion, minimal peak prominence, dynamic thresholding, and 138 vibration elimination and leverages them to solve problems 139 like over-counting, under-counting, and false walking which 140 can be encountered by the standard method. The method-141 ology was tested on a peak detection algorithm. It is also 142 common to use fast Fourier transform (FFT) to detect and 143 count steps [32], [33]. 144 Works focusing on heading estimation usually propose a 145 mathematical model that fuses the fetched data from IMU 146 units to determine the device's 3D orientation via either 147 Euler's angles or quaternions. The fusion process usually 148 involves Kalman filters, and sometimes quaternions. The next 149 step is to use the orientation of the device along with the data 150 from the IMU unit to estimate the heading of the user [34], 151 [35], [36]. It is important to notice, though, that smartphones 152 readily provide the pitch, roll, and azimuth of the device using 153 a virtual sensor called the orientation sensor. This sensor is 154 created using a fusion process similar to the one described 155 above [37]. 156

157
While the above mentioned works utilize step counting and 158 heading estimation techniques to detect relatively simple 159 VOLUME 10, 2022 actions like walking and running, the main contribution of this 160 work is to use the path found by applying PDR to recognize 161 more complicated actions, namely, Tawaf and Sa'i. In this 162 work, we design a system that is implemented on a mobile 163 application. 1  ing, a peak detection method is followed after fetching and 181 filtering the accelerometer data. A similar approach can be 182 found in [38]. For heading estimation, the output of the orien-183 tation sensor will be directly used, along with accelerometer 184 correction. The role of PDR in this paper is to provide a path 185 that can be appreciated by the detection algorithm. Therefore, 186 aiming for a robust, novel, PDR algorithm is not one of the 187 objectives of this work.  The rest of the paper is organized as follows. Section II 192 introduces the system framework and explains the underlying 193 processes that include pre-processing of sensors' readings 194 (Section II-A), path tracking using PDR (Section II-B), and 195 real-time detection and monitoring (Section II-C). Section III 196 provides results analysis for the different stages in the pro-197 posed model. Finally, the paper is concluded in Section V. 199 We propose a 3-stage system as shown in Fig. 1. The inputs to  In the first stage, the data is pre-processed for the PDR 207 algorithm. A standard coordinate rotation technique is used 208 for converting the given data from the device coordinate 209 system (DCS) to the global coordinate system (GCS). 210 1 We expect the pilgrim to carry a smartphone device with a pre-installed software application. The used smartphone must include a GPS and an IMU to operate the software properly. In the second stage, the PDR technique is used to infer the 211 user's path from the pre-processed data by step counting and 212 heading estimation.

II. SYSTEM FRAMEWORK
Step counting, which is mainly inferred 213 from the accelerometer, is used for step detection and stride 214 length estimation. As for heading estimation, it is mainly 215 inferred from the orientation sensor.

216
The final stage of the proposed framework is to detect 217 the users' activity and provide them with useful information 218 based on their location and situation. used for conversion between the two systems is given by where θ and φ denote pitch and roll angles, respectively.

233
It should be noticed that (1)   given by where N t and E t are the current northing and easting, N t−1 259 and E t−1 are the previous coordinates, SL t−1 is the length of  Step counting is the main process in PDR systems [24], along  A possible approach for step detection is to monitor the 271 smartphone's acceleration [24], [42]. Regardless of the phone 272 position relative to the user, each step has a significant con-273 tribution to the acceleration magnitude, given by Using this approach for step detection, however, yields 276 erroneous results due to two sources of distortion.  The ultimate goal of step counting is to estimate the crossed 299 distance by the user and draw his path, which will be used 300 accordingly in the detection algorithm. In this work, the 301 Weinberg formula given by  that is set depending on the user's step size. In our study, we 312 use two different approaches to tune the value of K . The first 313 approach is to train the tuning process based on GPS data.

314
If GPS data is not available, as in an indoor environment, 315 we set the step length to its average (73.6 cm [43]) and train 316 the K -factor so that the resultant average step length matches 317 this value. The sum of step lengths during the desired duration 318 results in the distance crossed by the user, which will be 319 combined with the heading angle (i.e., the last stage of PDR) 320 to draw the user's path.

321
Although distance estimation is needed for path tracking, 322 accurate estimations are not required for a valid detection of 323 the rituals as will be discussed in II-C. Therefore, even if 324 the step length was chosen to be constant, the algorithm is 325 still functioning. In other words, the output path will have the 326 same shape but will be scaled according to the K -factor and 327 will not affect the detection performance. Albeit random, typical smartphone settings and orienta-335 tions during Umrah activities can be predicted. Two common 336 settings are depicted in Fig. 5a and 5b. The front bag setting 337 in Fig. 5c is similar to the front belt setting in Fig. 5a in 338 terms of the pilgrim's heading analysis. Ideally, there are two 339 possible phone orientations for every setting in Fig. 5. If the 340 longitudinal side of the phone is perpendicular to the xy-plane 341 of the GCS (See Fig. 2b), then the phone position is vertical. 342 If the longitudinal side of the phone is parallel to the xy-plane 343 of the GCS, then the phone position is horizontal.

344
Another common setting for the phone is when it is held 345 in front of the user by his hand. We consider two possible 346 phone positions for this setting as well. One position is the 347 portrait mode, which is similar to the vertical position, and 348 the second position is the landscape mode, which is similar 349 to the horizontal position.

350
Considering these pre-defined positions for the smart-351 phone while performing the rituals and modifying the 352 heading estimation accordingly; will ease the process of the 353 heading correction significantly compared to the random held 354 situation, and hence, reduce the complexity of the model and 355 the real-time processing.

356
To successfully estimate the pilgrim's heading with respect 357 to his smartphone, it is first required to detect the smartphone 358 orientation using the uncompensated acceleration compo-359 nents. Smartphone orientation is determined by identifying 360 the component dominated by the acceleration of gravity. 361 Compared to the other acceleration components, the target 362 component, a i , is their maximum. In other words, we seek a i 363 such that where {a x , a y , a z } is a set of the uncompensated components 366 of the fetched vector, a DCS . Fig. 6 shows the acceleration for 367 three different cases; vertical, horizontal, and on the hand.

368
After determining the gravity acceleration component, 369 a i , tilt compensation is applied to the acceleration vector. 370 Fig. 7b and 7c illustrate the corrected acceleration direc-371 tions of the phone that correspond to the settings depicted 372 98236 VOLUME 10, 2022  in Fig. 5a and 5b, respectively. An angle is assigned to align where θ di is the angle of deviation on a certain axis 383 as illustrated in Fig. 7, a x and a y are the corrected 384 acceleration x and y components, respectively, and a i is the 385 deviated acceleration component obtained by (6).

386
The angle, θ di , is added to the ideal angle values to correct 387 the heading of the device. The total correction angle for every 388 situation of the device is in Table 1. The acceleration vector, 389 on which the complete heading estimation process operates, 390 is obtained by averaging acceleration values received during 391 the time taken for one step.

392
In case the smartphone was held in a way that was not 393 identified by the system, the user is notified to adjust his 394 phone in one of the positions as seen in Fig. 5.

396
After obtaining the path of the user using the PDR technique, 397 we design algorithms to detect and monitor the activity of 398 the user. In this section, a detailed explanation of the ritual 399 detection algorithms will be provided. For the first ritual, Tawaf, an algorithm was built to differenti-402 ate between Tawaf performing and non-performing pilgrims. 403 Detecting Tawaf movements allows providing feedback to 404 the user based on their current lap count and position. The 405 algorithm depends mainly on the angle with respect to the 406 Kaaba center for detection. Which is the center of rotation for 407 the Tawaf ritual. The Kaaba center is also used as a reference 408 in our detection as it is a fixed point with known coordinates. 409 Furthermore, several other conditions are used to check for 410 the integrity of the detection to avoid errors as will be shown. 411 VOLUME 10, 2022 The inputs to the algorithm are GPS and IMU sensors' data.  −π/4 rads with respect to the north which is the location of 435 Al-Hajar Al-Aswad which is where Muslims start the ritual 436 as indicated by the Tawaf line in Fig. 9).

437
After that, the algorithm starts monitoring, detecting, and 438 counting the laps. For this part of the algorithm, the Tawaf 439 area is divided into 5 regions, where every region occupies 440 one-quarter of the complete area except for the quarter con-441 taining the starting line. This quarter is divided into two 442 in order to create a region barrier at the start/end line of 443 each lap to detect lap endings. This is because completion 444 of the previous region is used as a condition to continue the 445 algorithm. Fig. 9 labels all the mentioned regions.

446
The algorithm monitors the process by continuously col-447 lecting data samples (i.e., pilgrim's position with time). The 448 algorithm ensures that each sample complies with certain 449 conditions to make sure the pilgrim is still performing Tawaf. 450 The main condition involves the angle, through which the 451 algorithm checks if the pilgrim is in the same region as the 452 one indicated by the previous sample. If the pilgrim is not 453 in the current region, the algorithm checks if the pilgrim is 454 in the next/following region. If this is the case, the algorithm 455 checks that the number of samples taken during the previous 456 region is reasonable enough to expect a region transition. This 457 serves as the second condition which is a time restriction 458 where the pilgrim must have spent enough time/samples in 459 the current region for it to be considered a completed region. 460 The third condition is the radius condition, for which the 461 pilgrim must stay within a certain radius or this will cause 462 the algorithm to stop. If any of the conditions to continue the 463 algorithm are broken, then this prompts the pilgrim to check 464 whether they plan to continue, stop, or take a break and then 465 continue the ritual. These actions will go on where the region 466 counter resets every lap, and the algorithm stops when the 467 lap counter reaches 7 (i.e., the pilgrim has finished the Tawaf 468 ritual). To accommodate random errors, a success percentage 469 is introduced as a minimum requirement for the number of 470 samples that need to be correct for proper region transition. 471 The success percentage is currently set to 90%, but it can be 472 adjusted in real-time on-site testing for extreme cases. After Tawaf, pilgrims move to the Sa'i (Masa'a) area to per-475 form the Sa'i ritual (see Fig. 9). An algorithm is designed to 476 detect the Sa'i and provide useful information to the pilgrim 477 based on his activity. First, the algorithm checks if the pilgrim 478 enters the Masa'a area and starts Sa'i by analyzing the path. 479 Once the pilgrim crosses more than 100 meters, in the 480 direction of Al-Marwa (Fig. 9) Fig. 11a includes three paths, the GPS path in blue, the PDR 506 path before the angle correction in dotted yellow, and the 507 path using the same technique with angle correction in red. 508 Fig. 11b and Fig. 11c show two paths of the user while carry-509 ing the phone horizontally to the belt as in the situation (b). 510 As discussed in Section II-B, the path detection algorithm 511 monitors the phone position and updates it with each step. 512 Fig. 11d shows the output path in case a transition between 513 different phone positions occurs. First, the user was walking 514 in a straight line while keeping the phone in his pocket, then 515 he/she held it by hand in portrait mode. Fig. 11e shows a 516 simple correction for the heading angle in the case of holding 517 the phone by hand but in a tilted position.

518
To examine the reliability of the extracted PDR path, 519 we test its accuracy by comparing it with the GPS data as 520 a reference. Table. 2 shows the average distance deviation 521 between the PDR and the GPS paths for different distances 522 and phone positions. Results show that as the distance trav-523 eled increases, the deviation between the paths becomes 524 larger due to the accumulation of error. This can be improved 525 by updating the location regularly using GPS. We also notice 526 that the phone position is a major source of error. Unlike 527 holding the phone by hand, putting it in the pocket or holding 528 it to the belt makes it more vulnerable to vibrations, there-529 fore, accuracy reduction is expected even with the proposed 530 heading correction approach.

532
To verify the Tawaf detection algorithm, it was tested using a 533 real path as shown in Fig. 12a. The location illustrated in the 534 figure was chosen for its similarity to the Tawaf area. Also, 535 VOLUME 10, 2022  will not be counted as a part of Tawaf which is why it is 565 shown as the pre-Tawaf path in the figures below. This also 566 confirms that if a pilgrim did not start Tawaf at the correct 567 starting angle range, the first lap will not count because it was 568 not complete. The path in the test was completed at the end 569 with 7 complete laps to confirm a complete Tawaf detection. 570 This link 2 shows a video demonstrating the algorithm running 571 and detecting the ritual from the path. The PDR path is also 572 very smooth compared to the path from the GPS. This is 573 again due to the high sampling rate used in the IMU sensors 574 which is not possible for the GPS sensor. The algorithm also 575 handles any errors that break the Tawaf continuity conditions 576 (e.g. going in the reverse direction) by pausing the process 577 and waiting for the pilgrim's response. If the response is to 578 continue the data collection will continue from the last correct 579 sample. If there is no response, or the response was to stop the 580 algorithm, then the algorithm will reset.

581
Although results for Tawaf detection are satisfactory in the 582 experiment conditions, further testing is needed to see the 583 performance in real Tawaf conditions. This is due to many 584 factors such as the different geographical locations and the 585 large number of mobile phones in the area which could affect 586 the GPS data. Also, the two methods (GPS and PDR) need 587 to be tested independently to study their effect on the battery 588 life. Notably, the PDR curve is very smooth compared to the 589 GPS curve due to its high sampling. However, as both paths 590 were successfully detected, it might be worth lowering the 591    Fig. 13b, where the PDR path is plotted in red 604 and the GPS path is plotted in blue.

605
As illustrated in Fig. 10, the algorithm considers time, 606 distance, and heading direction to count the lap. Therefore, 607 unless the user starts from the correct starting point of the 608 Sa'i ritual, labeled as Safa in Fig. 9, the algorithm will not 609 be triggered, and laps will not be counted. On the other hand, 610 when a user performs extra laps (i.e., more than seven laps), 611 which is the case in Fig. 13b, the algorithm terminates the 612 activity and sends a message that the Sa'i ritual is completed. 613 Considering the similarity between the two paths in 614 Fig. 13b, the PDR-based tracking approach is successful. 615 It can be noticed that there are also some synchronization 616 issues due to, as discussed in III-B, the difference in sampling 617 rates between GPS and the IMU unit. It can be noticed that 618 the PDR path is smoother than the GPS path due to the high 619 sampling rate for the IMU unit.

621
One major limitation to the process explained in II-C is the 622 need for periodically collecting background GPS data before 623 triggering the detection algorithm. The sampling rate of GPS 624 is required to be high enough to detect the start of the ritual. 625 Otherwise, the algorithm may not be triggered and, conse-626 quently, may not be able to perform its task correctly. Back-627 ground collection of the GPS data could affect the battery life 628 of the phone, especially at high sampling rates. More analysis 629 is needed to determine the optimal period of background GPS 630 data collection to automatically detect the starting point of the 631 ritual and minimize battery utilization. Also, if background 632 data collection is disabled, the detection process will require 633 user activation.

634
Another limitation is imposed by the algorithm's depen-635 dency on specific phone positions. Phone positions that are 636 not considered in II-B3 may occur, albeit rarely, due to 637 unexpected situations like falling and sliding. The algorithm 638 handles such abnormalities by requesting the user to adjust 639 their device, which can be inconvenient for the user. Another 640 situation where user's input is required is when they decide 641 to take a break. If the algorithm is not informed directly by 642 the user that Tawaf or Sa'i is stopped, the algorithm stops 643 tracking and then restarts at the beginning of the next lap. 644 In this case, the detection process will be incomplete, since 645 the pilgrim needs to complete the current lap when they 646 return to the ritual. Therefore, a better error handling process 647 that enables the algorithm to automatically recover when the 648 pilgrim returns to the correct path would be beneficial in the 649 future.

650
Finally, the proposed algorithm naturally inherits the limi-651 tations of the employed tracking algorithm. The main draw-652 back of PDR is error accumulation in its extracted path points 653 (see Section III-A). As a solution, the use of GPS data to 654 correct the paths regularly is proposed. However, GPS access 655 is not always available during the rituals, hence, the system 656 is more vulnerable to making errors in indoor areas like the 657 Sa'i area.   Sensing. He has authored two book chapters and more than 130 articles. 883 His research interests include direction of arrival estimation, throughwall-884 imaging, localization, channel characterization, and ultra-wideband signal 885 processing. He was a recipient of many awards in the excellence in teaching, 886 advising, and instructional technology.