An AI-in-Loop Fuzzy-Control Technique for UAV’s Stabilization and Landing

In this paper, an adaptable fuzzy control mechanism for an Unmanned Aerial Vehicle (UAV) to manipulate its mechanical actuators is provided. The mission (landing) for the UAV is defined to track (land on) an object that is detected by a deep learning object detection algorithm. The inputs of the controller are the location and speed of the UAV that have been calculated based on the location of the detected object. Two separate fuzzy controllers are proposed to control the UAV’s motor throttle and its roll and pitch over the mission and landing time. Fuzzy logic controller (FLC) is an intelligent controller that can be used to compensate for the non-linearity behaviour of the UAV by designing a specific fuzzy rule base. These rules will be utilized to adjust the control parameters during the mission and landing period in runtime. To add the effect of the ground for tuning the FLC membership function over the landing operation, a computational flow dynamic (CFD) modeling has been investigated. The proposed techniques is evaluated on MATLAB/Simulink simulation platform and real environment. Statistical analysis of the UAV location reported during stabilization and landing process, on both simulation and real platform, show that the proposed technique outperforms the similar state-of-art control techniques for both mission and landing control.

ronmental noise and during the mission and landing period is 23 still a challenge [3]. 24 For a smooth, fast, and reliable UAV movement, regardless 25 of the perceptual sensors that has been used to control it, 26 such as GPS [4], optical flow [5], ultrasonic sensors [6] 27 and laser range finder [7], the controller should manipulate 28 the actuators in different environmental situations during the 29 The associate editor coordinating the review of this manuscript and approving it for publication was Zhongyi Guo . mission and landing periods. These environmental conditions 30 are accompanied by noise and in some cases, such as the 31 period of landing the UAV, affects the dynamics of the UAV 32 and must be considered in control design in runtime. On top 33 of this all the computing and mechanical process demand 34 onboard power resource that, in the case of UAVs, is the 35 mounted battery. Even though there exists several control 36 techniques for UAV controls, still there exists a unified light- 37 weigh control technique that can be adopted based on differ-38 ent environmental conditions in runtime. 39 Another issue is the perceptual sensors to detect the suit- 40 able landing place that might be even mobile. Using GPS 41 data to detect the landing place removes significant portion of 42 onboard power consumption for object detection [4]; however 43 it is noisy and limits the operation of the UAV on to those 44 areas that GPS is available [8]. Infrared based-approach allow 45 to estimate the pose of a UAV by using infrared lamps and 46

100
• Development of both simulation and real platform to 101 evaluate the proposed idea and comparing the results 102 with state-of-the-art. 103 The remaining part of this paper is divided as follows: 104 Section II reviews the related work about the developed 105 control systems based on external environment and internal 106 environment. System overview is illustrated in III. Section IV 107 explains the algorithms used in the developed system in 108 detail. The experimental setup is covered in Section V. 109 Simulation and real platform results are demonstrated in VI 110 followed by the conclusion in Section VII. 112 Recent advancements in both fields of UAV control system 113 and artificial intelligence (AI) have made a certain realm 114 of vision-based autonomous applications possible [11], [12], 115 [13], [14]. The operation of a fully automated UAV system 116 can be divided into three working periods namely 1) take off 117 period 2) the mission period, i.e., tasks to be performed, and 118 2) the landing period. Among these three periods, efficient 119 and robust control techniques for UAV stabilization over the 120 mission and landing period is an important necessity, since 121 the UAV must full-fill several requirements such as smooth 122 path trajectory, high stability, and smooth and relax landing 123 to decrease the damage on both cargo, in the case of, e.g., 124 delivery mission, and physical parts of the UAV [15]. 125 In [16], the authors proposed a control system for sta-126 bilizing the UAV that is following a moving object under 127 various speeds. To do this, a Gain-Scheduled PID controller 128 (GPID) has been adopted that refines the system's actuation 129 based on the received feedback. Here the actuation are the 130 motor thrust and the feedback is the location of the object 131 under track. However, in the proposed idea, due to linear 132 relationship of input feedback and output actuation values 133 of the GPID controller, the system is showing a non robust 134 behaviour while big disturbances in the input is experienced. 135 In [17], the authors proposed a feed-forward proportional 136 integral (PI) stabilizer for pUAV to solve the same problem, 137 thereby disturbance parameters directly affect the control 138 parameters in stabilization while tracking. However, since 139 calculating the control parameters demands heavy workload, 140 mainly such techniques require an offloading process through 141 which the heavy calculation of the parameters happens in 142 stationary resources. This highly affects the response time of 143 the controller and makes it infeasible for UAVs to operate in 144 areas without network connection.

145
The popular PX4 autopilot software which is employed in 146 UAV applications for stabilization and position control [18] is 147 based on two nested loops, i.e., a proportional gain, P, in the 148 outer loop, and PID in the inner loop. The outer is responsible 149 for controlling and stabilizing the UAV orientation angles 150 while the inner loop is responsible for controlling the angular 151 velocity of the UAV. However, because of the fixed gain of the 152 proportional gain of the inner and outer controller loops, the 153 controller behaviour is not robust while encountering external 154 disturbances, e.g., wind, which will require continues change 155 in the PID gains.  In light weight techniques, even though the computational 210 cost is low and the control system is energy efficient, the 211 non-linearity behaviour of the UAV functionality and the 212 environmental conditions, makes the control system non-213 robust and untrustable, especially under the effect of high 214 disturbances. In contrast, in heavy learning techniques, even 215 though the non-linear behaviours of the system models is 216 considered, run-time data processing and transfer causes huge 217 computation and communication cost that affects the energy 218 efficiency and response time of the control unit. Another 219 important fact in designing such control techniques is the non-220 probabilistic behaviour of the UAV physics and environment 221 that makes the possible stochastic methods practically impos-222 sible leading to a fuzzy solution for such a modeling, i.e., not-223 probabilistic.

224
Overall, in this paper a fuzzy technique is proposed to 225 control the UAV over the mission and landing period. Gener-226 ally, the FLC requires enormous computation which increase 227 the computational burden of the system [28], [29] and to 228 overcome this, there are some developed methods to reduce 229 its execution time. Authors in [30] proposed a technique to 230 simplify the FLC rule base table for induction motor drives. 231 Their results show that the simplified rule base has similar 232 result to the standard rule base and is less computational. 233 Another approach to accelerate the FLC computation process 234 is using a lookup  In this technique, a non-probabilistic, non-linear member-240 ship function monitors the influential parameters over the 241 mission and landing period upon which adjusts the control 242 parameters at run-time. In this method, there exists different 243 modes of control each of which provides different parameters 244 for the control system to stabilize the X -Y location over 245 the mission period and X -Y -Z location of the UAV over 246 the landing period. Two fuzzy controllers for the mission 247 and landing period are proposed that are called stabiliza-248 tion and landing controllers respectively. The stabilization 249 controller during the mission and landing period while the 250 landing controller only operates during the landing period. 251 The membership function for these two fuzzy controllers are 252 based on different behaviors the UAV shows over the mission 253 VOLUME 10, 2022 3) stabilization, and 4) flight controller, thereof flight con-295 troller is executing on the low-performance processor and 296 the others are executing on high-performance processor (it is 297 distinguished by different block colors in the figure).

298
Landing algorithm consists of distance estimation algo-299 rithm and the landing controller part. The input of distance 300 estimation algorithm is the location of the detected object 301 in the image that is represented as an boundary box on the 302 image. This function first calculates the distance between 303 the UAV and the object, then it calculates the altitude of the 304 UAV based on this calculated distance. Using the 2D cam-305 era gimbal the camera always points downward that makes 306 the distance estimation algorithm possible to calculate the 307 altitude, i.e., 'Z ' of the UAV, needed for landing control. 308 The calculated altitude, i.e., 'Z ' value in Figure 2, will be 309 subtracted from the target altitude Z ref to calculate the 'Z ' 310 error to be corrected in the landing controller. The output of 311 the landing controller is the throttle adjustment that will be 312 passed to the flight controller unit.

313
Stabilization algorithm consists of object location calcula-314 tor and a stabilization controller. The input of object location 315 calculator is the location of the detected object in the captured 316 image. Based on this location, the 'X ' and 'Y ' location of the 317 UAV will be calculated. Then, these values will be subtracted 318 from the reference 'X ref ' and 'Y ref ' preparing the error to be 319 corrected in stabilization control algorithm. The output of the 320 stabilization control algorithm are the roll/pitch angles that 321 will be passed to the flight controller.

322
In the following each control algorithm will be discussed 323 in more details. After receiving the captured image from the camera, a convo-327 lutional neural network (CNN) algorithm is applied for object 328 detection. In this work, a MobileNet object detector is used 329 that is a modified version of Single Shot Multibox (SSD) 330 object detection algorithm [38].   convolutional neural networks that are vastly smaller in size 336 and faster in performance than many other popular models.

337
Depth-wise separate convolutions first apply a single filter 338 on each input to filter the input data, followed by 1 × 1 con-339 volutions which combine these filters into a set of output   [40], and data-based empirical models have also been 368 devised for multi-rotor control [41]. In the present work,  Figure 4). The distance from the two rotors to 386 the symmetry plane is 272.5 mm and 232.5 mm, respectively, 387 and they are 514.1 mm apart from each other. The rotor blades 388 were modeled in ANSYS SpaceClaim using measurements 389 from the UAV.

390
ANSYS Fluent 2019R3 was used for both creating the 391 spatial discretization (mesh) and for solving the Reynolds-392 Averaged Navier-Stokes equations in the steady state. The 393 polyhedral mesh used in the solution consists of roughly 394 115000 volume cells, with an appropriate boundary layer 395 structure on the walls (cf. Figure 6). Air was treated as 396 incompressible turbulent gas, using the k − ω SST turbulence 397 specification. The boundary and cell zone conditions used in 398 the CFD simulations are listed in Table 1.

399
The CFD simulation results consist of the air velocity com-400 ponents and static pressure values at all points in the domain. 401 Figure 5 shows a comparison of the predicted flow velocity 402 magnitudes at propeller midsection planes for A = 50 mm 403 and A = 1500 mm. Here, f rot = 4000 rpm, which is the max-404 imum rated rotation speed for the propellers. Clearly the flow 405 fields induced by the rotors are intertwined in near-ground 406 conditions, whereas the rotors operate more independently 407 away from the ground.

408
The upward lift on the propellers, as predicted by CFD, 409 consists of pressure and viscous components. As expected, 410 the CFD results show that the maximum lift is seen in near-411 ground conditions (A = 50 mm) and at the highest rotation 412 speed (f rot = 4000 rpm). Figure 7 displays the predicted 413 lift in relation to that maximum force, for several altitudes 414 and rotation speeds. It is clear that the ground effect prevails 415 VOLUME 10, 2022   pinhole camera.
where f is the focal length of the camera, r radius of the 427 marker in the image plane, R radius of the marker in the object 428 plane, and d distance from camera to the object in (cm). From 429 the previous equations, d is calculated by using trigonometry 430 as demonstrated in Figure 8. After calculating the error between the Z and Z ref , the error 433 value is fed to the landing control as shown in Figure 2 to 434 give out the suitable throttle adjustment value that is required 435 to land the UAV safely on the detected object.

436
To account for the ground effect that exists near the ground 437 while the UAV is descending, FLC is utilized to control the 438 UAV landing. In the present work, the UAV lifting force is 439 assumed to be divided into 1) UG (Under ground effect), 440 and 2) OG (Outside ground effect) conditions. Here, UG is 441 a condition where, due to the ground effect, less power is 442 required, and OG refers to conditions other than UG.

443
The definition of a fuzzy set permits one to assign values 444 to fuzzy variables as inputs and outputs. The input/output 445  • If the UAV is in the OG range, a normal landing algo-489 rithm will be applied to maintain a fixed speed while 490 landing (−0.125m/s).

491
• When the UAV reaches the UG region, a special speed 492 control algorithm based on the position of the UAV will 493 be triggered to overcome the effect of the ground effect 494 while landing.

495
• The FLC checks the moving direction of the UAV, 496 in which it will send a high negative throttle adjustment 497 values if it was moving upwards in order to change its 498 direction.

499
• If landing speed increases or decreases, the FLC will use 500 the throttle adjustment to make it constant.

501
The rule base shown in Table 2, works in two different 502 modes based on the UAV distance, Mode A and Mode B.

503
Mode A is the normal operation mode of the UAV when it is 504 outside the UG region, i.e., the movement is mainly horizon-505 tal and the view shouldn't be focused in one specific object. 506 Moreover in this mode there is not any effect of the ground, 507 as an external environmental effect, on the controller system. 508 In this mode, the throttle adjustment values work between NS 509 to PS to keep a constant landing speed of −0.125m/s, until 510 any sudden change in speed or direction of the UAV occurs 511 where it will generate a throttle adjustment corresponding to 512 this change (NM to PM).

513
Mode B is the second scenario where the UAV is within the 514 UG range. Through this situation, the internal and external 515 state changes and the environmental information is mainly 516 high, due to focus on specific area, and the speed is low. 517 In this region, the throttle adjustment output is shifted to 518 higher gains values according to the UAV position whether 519 it's very close to the ground (UG_NG) or far from ground 520 (UG_FG). In the UG_FG, the throttle adjustment is shifted 521 from (NS-PS) to (NM-PM) to overcome the ground effect 522 and constrain the landing speed of -0.125m/s. In the UG_NG, 523 the ground effect gets more effective which will lead to the 524 decrease of landing velocity until the UAV keeps hovering 525 VOLUME 10, 2022 above the ground without being able to land. To overcome 526 this problem, the throttle adjustment is shifted again to higher In Figure 2, once the object is detected, its position is given to 548 the object stabilization block to start the stabilization process.

549
In this work, a Fuzzy-PID controller is utilized to overcome  from −20 to 20. The fuzzy linguistic variables of the inputs 574 are defined as NB, NS, ZE (Zero), PS, and PB. Figure 11(b) 575 specifies the output of Fuzzy-PID (K pf , K if and K df ) that is 576 added or subtracted to the fixed gains K p , K i and K d , as illus-577 trated in Equations 7-9, and its linguistic variables are defined 578 as NB, NS, ZE (Zero), PS, and PB. K p , K i and K d values are 579 obtained from studying the effect of these parameters on the 580 simulation model that is covered in Section V.B.2, and was 581 further explained in our previous work [20]. The G value is 582 by default '1', however for the sake of simplicity in tuning 583 the Fuzzy-PID without the need to modify its membership 584 functions, the G value can be modified to tune different 585 systems.

587
Designing the rule-base for the Fuzzy-PID is the most crucial 588 part because the whole function of this controller depends 589 on it. For this, several experiments were conducted in addition 590 to experience to design it. Table 3-5 show the rule base that is 591 used to utilize the P, I and D gains of the PID controller for 592 the UAV stabilization.

593
The principles of constructing the Fuzzy-PID rule base are 594 as follow:

595
• When pos is NB/PB (UAV is far from the center of the 596 detected object) and the UAV is moving away from the 597 detected object (e.g., pos is NB and pos is also NB), 598 K pf should be big to increase the P gain of the PID to 599 push the UAV towards the center of the detected object. 600 On the other hand, K df and K if ought to be NB, so that I and D become very small for better following perfor-602 mance and to avoid any oscillation.   In this work, a UAV multicopter system is demonstrated 628 as shown in Figure 12. The multicopter consists of 629 FIGURE 12. UAV multicopter practical system. a high-performance CPU, Jetson TX2, installed on a J120 630 carrier board, powered with a 3S 11.1V LiPo battery that 631 weights 120g and is connected to an ultra HD Logitech 632 webcam through USB. The camera is attached to a 2D-gimbal 633 that is used to make toe camera face the ground while the 634 multicopter is landing. The proposed system also consist of 635 a low-performance CPU, Pixhawk 2.4.8 flight controller, that 636 is based on a 180 MHz ARM R Cortex R M4, and is equipped 637 with a 10DoF IMU, eight PWM outputs, four UART and 638 other different connections. The flight controller is build on 639 an open source program ''ArduPilot'', which is codded in 640 C++. Furthermore, it can be easily monitored or config-641 ured from a ground control station using an RF transmit-642 ter/receiver. The low-performance CPU is connected to the 643 high-performance CPU using a UART communication.

681
For evaluating the proposed technique, MATLAB/Simulink 682 simulation environment has been used [44]. Figure 14 shows 683 the Simulink models of the multicopter platform. Within 684 this simulation, multicopter parameters, e.g., initial location, 685 weight, size, etc., can be changed. The location error is calcu-686 lated by taking the difference between the multicopter current 687 position (X , Y , and Z ) and the reference position (X ref , Y ref , 688 and Z ref ). Afterwards, the error is passed to the proposed 689 stabilization and landing control blocks thereafter the attitude 690 commands (Roll/Pitch/Throttle) will be passed to the attitude 691 controller block and the multicopter control mixing block, 692 which are responsible to convert the attitude commands to 693 actuation signals to the motors. The multicopter dynamics 694 consists of mathematical equations about the dynamics of 695 different parts of the multicopter. More details about these 696 equations can be found in [45].   the K , the faster the landing speed. However, when K is very 714 high, it will cause bumping leading to the damage of the 715 multicopter. Therefore, a proper K should be selected.

716
A landing comparison between the proper selected param-717 eters of PID and FLC is shown in Figure 15(c). The total time 718 for landing using the PID controller is 60 sec, while FLC 719 takes 40 sec. Due to the ground effect, the PID has slower 720 response while approaching the ground (UG range) due to its 721 fixed gains. On contrast, the FLC shows better response when 722 landing due to its rule base which combines both speed and 723 distance, that will ensure a smooth and safe landing process. The average error for X and Y are approximately zero for 765 all the control configurations. This is because of the random 766 disturbances that have been injected in different direction on 767 X -Y plane. However, the MAX and STD values show that the 768 proposed technique can significantly reduce the disturbance 769 error in X and Y dimensions. This can be noticed for both 770 configurations with our proposed stabilization, i.e., L-PID + 771 S-ours and L-ours + S-ours. In Z dimension, the landing 772 algorithm shows its robustness by reducing the MAX and 773 STD values of the error comparing to the other techniques.

774
It can be seen that the stabilization part improves the 775 MAX and STD of error while working with the proposed 776 landing controls, i.e., L-ours + S-ours, comparing to L-PID + 777 S-ours and L-ours + S-GPID, and L-ours + S-FLC. This 778 shows that the proposed landing and stabilization algorithm 779 give the best outcome when both techniques are working 780 together as shown in Figure 17. This is due to the fact that 781 the command signals to the propellers are the ensemble of 782 both controller commands that are merged and tuned together 783 to manipulate the mechanical parts of the multicopter, such 784 as propellers and other mechanical parts. Any change or 785 disturbance of the multicopter location in any axes affects its 786 location in other axes since the multicopter control mixing 787 unit converts the required attitude corrections into throttle 788 percentages for each motor. The higher the attitude correc-789 tions values, the more throttle given to each motor, which 790 in return will lead to more disturbances of the multicopter 791 during landing. 792 Figure 19 shows the landing time of the 6 different 793 controllers. Using the full-fledged approach, i.e., L-ours + 794 S-ours, will reduces the operation time by 6%, 8%, 51%, 795 59%, and 65% comparing to (L-ours + S-FLC, L-ours + 796 S-GPID, L-PID + S-ours, L-PID + S-FLC, and L-PID + 797 S-GPID) respectively. The reason is that by reducing the 798 amount of disturbances and providing more stability, the 799 multicopter can reach the target in a faster and smoother way 800 comparing to the other techniques.  Figures 20(a-c) show the X , Y , and Z positions of the 803 multicopter while using different control techniques in our 804 experiment on a real environment. Figure 22 shows the cor-805 responding statistical values about the occurred error in X , Y , 806 and Z axes, i.e., the standard deviation (STD), maximum error 807 (MAX), and average error (AVG), while Table 7 shows the 808 average statistical values of the designed techniques during 809 the landing and stabilization process throughout the experi-810 mental period. As it can be seen, opposite to the simulation 811 results the average disturbance in X and Y direction is not 812

818
L-PID + S-GPID shows the poorest response among all 819 the tested techniques. Although the S-GPID is easy to imple-820 ment, entailing a fast way to control the system, it will 821 fail to control the multicopter in case of high disturbances, 822 especially when combined with L-PID because of its linear-823 ity which will require continuous change of the PID gains. 824 On the other hand, S-FLC have better outcome compared 825 to S-GPID because this controller considers both the error 826 value and its derivative as the inputs targeting to minimize 827 the error between the reference and the actual output. Since 828 the proposed Fuzzy-PID controller S-ours auto-tunes the PID 829 gains, according to its two inputs 1) position and 2) change 830 of the position of the multicopter, it can rapidly reduce 831 the error between the multicopter and center of target with 832  The object detection part takes a large execution time due 871 to its huge data size, and implementing two other algo-872 rithms, e.g., FLC and Fuzzy-PID controllers, in parallel with 873 the object detection will cause processing delay. For this, 874 MATLAB/Simulink is used to generate a lookup table that 875 calculates the output of the FLC and Fuzzy-PID controllers 876 corresponding to different input values instead of the real-877 time FLC. Table 8 shows the average time for executing 878 the three algorithms; object detection, landing, and stabiliza-879 tion, of the designed controllers. As can be seen in Table 8, 880 due to the usage of lookup table, the computation time to 881 execute a FLC and a Fuzzy-PID is slightly larger than a 882 Gain-Scheduled PID controller in the stabilization algorithm 883 (average 0.35 msec and 2.35 msec respectively), and the 884 difference between a FLC and a PID controller is 0.96 msec 885 on average in the landing process. Combining both timing, 886 in addition to the object detection algorithm and calculating 887 the total time of each designed technique, our proposed con-888 troller L − ours + S − ours execution time is slightly higher 889 and can be neglected in comparison to other techniques. 890 Furthermore, having a better stability, settling time, and more 891 energy efficient compared to the other approaches, makes the 892 proposed technique more suitable for such application.

894
In this paper, two fuzzy control techniques for stabilization 895 and landing are proposed that considers manipulating the 896 multicopter's roll and pitch. The mission (landing) for the 897 UAV is defined to track (land on) an object that is detected 898 by a deep learning object detection algorithm. The inputs of 899 the controller are the location and speed of the UAV that has 900 been calculated based on the location of the detected object. 901 Two fuzzy control techniques are proposed to control the sta-902 bilization of the UAV during the mission and to overcome the 903 ground effect during the landing period. Several experiments 904 are performed, and statistical analysis of the UAV location 905 are acquired. The obtained results show that the proposed 906 technique has low computation time, better performance, and 907 less energy consumption for both two operational modes of 908 mission and landing in comparison with other technique.

909
Future studies could improve the current work by studying 910 the effect of using a reinforcement learning technique to auto-911 tune the PID controllers. Furthermore, using an adaptive FLC 912 for landing process with a simplified rule base technique to 913 overcome the ground effect is also considered to improve 914 this work.