Cost-Effective Flight Strategy for Aerial Thermography Inspection of Photovoltaic Power Plants

Thermography from unmanned aerial vehicles (UAV) is widely used for module condition surveys and defect detection in solar (photovoltaic) power plants. This article presents an optimized defect inspection procedure in which a two-stage autonomous flight strategy is adopted to reduce the operation time, and thereby the cost. The first stage is a fast high-altitude flight for rapid coverage of the entire plant. The resolution on the modules is then somewhat degraded, but adequate for detection of possible defect locations. In the second stage, an optimized flight path is calculated to revisit and image only the detected locations from a lower altitude, where the resolution is sufficient for the classification of defects. This concept is studied through simulations of different plant geometries and defect densities. The simulations are supported by actual data, including a plant-scale survey. The proposed strategy is shown to have the potential for significant savings in operation time, on the order of 60% in the experimental case with 2% module defects. Commonly reported defect densities of 0.5% to 1% will give even larger savings. The inspection strategy is shown to be especially beneficial for plant geometries representative of high latitudes.

paradigms for operation and maintenance efficiency. Thermography is a robust and well-established inspection tool because deviations in the module temperature distribution is a strong indication of a defect [2], but manual thermal inspection entails significant labor cost for the utility-scale power plants of today. Recent advances in aerial technology, sensors, and control systems make unmanned aerial vehicles (UAVs) a natural tool for the inspection and monitoring of PV power plants. With an onboard thermal camera and a GNSS-referenced navigation system, thermal module anomalies can be efficiently detected and localized. Defects can then be classified according to their appearance: Active bypass diodes result in heating of one substring, a short-circuited module is recognized as a patchwork of hot cells, and external shadowing or cell cracking can make a single cell clearly warmer than the others. Heating of a part of a cell, or a point, is also observed [2]. Classification of different module defects can resolve weaknesses in the plant operation, module installation, or production. Classification is also essential to determine the severity of a failure, in terms of both power loss and safety. A bypassed substring results in a module power reduction of 33%, while loss due to a cell failure can be less than 1% of the module output [3]. On the other hand, a cell hotspot can reach temperatures above 100°C, and pose a risk to the safety of, e.g., rooftop PV systems.
A useful distinction is made between the detection and classification of module defects. Distinguishing between different types of defects for classification requires complex algorithms and high image quality. The module resolution requirements for defect classification constrain the UAV inspection altitude, and thereby the flight time. On the other hand, detecting a thermal anomaly, with no consideration of defect type, is a relatively simple task with only moderate requirements on processing and image quality. Consequently, the UAV altitude and flight speed can be increased, resulting in faster plant coverage. An efficient inspection flight is crucial for the cost-effectiveness of plant monitoring, as the main cost relates to the presence of trained personnel [4]. We, therefore, suggest a two-stage autonomous flight planning strategy involving an exploration stage to quickly detect possible module defects in the entire plant from a high altitude, and a classification stage revisiting only the detected anomalies, at a lower altitude, where resolution requirements for defect classification are met. We use a simulation to compare the described flight strategy with more traditional flight patterns This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ for different power plant geometries and defect densities. The findings of the simulation are supported by real data from our defect inspections. We show that significant savings are possible through reduction of the total duration of the operation. This work builds on [5], presented at IEEE PVSC 2021, supplemented by results from a new simulation model.

A. Path Planning
The two most common flight patterns for PV inspection are row-by-row flights, or to follow a "lawn mower" path, as illustrated in Fig. 3(a). The advantage of these flight patterns is the robustness and predictability of a predefined path. A more efficient flight can be achieved with path planning, an area of rapid development in the field of autonomous systems. This has briefly been explored in a PV inspection setting: Ding et al. [6] optimize the UAV path to minimize the number of energy costly turns in terrain with irregular distribution of PV strings, Kong et al. [7] have developed an infrared vision-based automatic navigation strategy. Both methods automate a row-by-row flight path. Henry et al. [8] have also developed a row-by-row path planning algorithm, while Sizkouhi et al. [9] automatically generate a lawn-mower flight path that covers an extracted power plant area. Sizkouhi et al. [10] present a method to extract the boundary of a PV plant using a convolutional network, which can facilitate further path planning. In [11], three different coverage path-planning methods are investigated for three different PV power plants, where the aim is to find the path with the least UAV energy consumption.

B. Defect Detection
Detecting defective modules is essential for PV plant maintenance and warranty assessments. Defect detection with aerial thermography has been done with various image processing techniques: Addabbo et al. [12] utilized a template matching algorithm to detect module defects, which are then geo-referenced. In [8], [13], [14], and [15], the detection algorithm is based on module intensity statistics within automatically extracted module boundaries. Binary thresholding is another well-tested method to segment hot regions within a PV module for defect detection [16], [17]. A water filling algorithm is deployed in [18], and recently also deep learning-based methods have been presented [19], [20], and [21]. The defect detection algorithms in [17] and [18] can be run in real time.

C. Defect Classification
An image resolution corresponding to 5×5 pixels per cell of 15×15 cm is considered a minimum for classification of module failures by The International Electrotechnical Commission (IEC) [22]. IEC presents 12 categories of thermal anomalies divided into module patterns, substring patterns, and local abnormalities (such as heated cells and points), all with different underlying causes. The International Energy Agency suggests eight categories of module failures based on the thermal appearance [23]. Grimaccia et al. [24] and Francesco et al. [25] perform a module recognition on image mosaicking of the PV plant, followed by a defect classification that recognizes cell hotspots, activated bypass diodes, and disconnected modules. A deep learning model is used by [26] to differentiate between four types of defects with up to 89.5% accuracy, while Li et al. [27] present a deep learning-based solution to identify five different defects from RGB images. The latter has a mean accuracy of 97.9% and uses a dataset of 8400 aerial images. In [28], a convolutional neural network (CNN) classifies six PV cell defects in the industrial production line with eddy current thermography. A fine-grained classification of ten thermal anomalies is achieved in [29], with a test accuracy of more than 90% using CNN and a dataset of 4.3 million images.

III. FLIGHT STRATEGY
Noting the efficiency potential in a flight pattern that only permits defect detection, we here explore a combined measurement strategy with two stages: In the first exploration stage, the entire plant is imaged with a fast flight at a high altitude, in a pre-defined lawn mower pattern such that multiple rows are imaged simultaneously. A higher velocity is acceptable because the increased altitude results in a lower observed ground speed, and therefore less motion blur. Images are processed to detect thermal anomalies, a process that can be performed in real time [17], [18]. The detections are automatically georeferenced [30]. Accurate georeferencing from a large altitude introduces challenges, as a small deviation between the measured and true camera orientation will propagate to large offsets on the ground. At 80 m, a mismatch of 1 degree will result in a 1.4-m ground offset for a camera pointing directly downwards. This is, however, acceptable, as it is well with the field of view during the revisit at the classification stage. A direct georeferencing from 80 m would in this case tag the adjacent module, so a more accurate georeferencing is performed from the classification stage.
In the second classification stage, the UAV follows an optimized path to revisit only the detected anomalies, at a lower altitude and slower speed to collect higher quality imagery for defect classification. The output from the classification determines if a module replacement is required, or if the thermal anomaly is caused by external factors such as shadowing from vegetation. Finding the shortest path that revisits all the detected anomalies once is a "traveling salesman problem." An optimal solution is infeasible for a high number of locations, however, in our case, an approximate solution provides the benefits we seek. Different heuristic approaches to the traveling salesman problem exist, and choosing an algorithm is a compromise between execution time and closeness to the optimal solution. Perhaps the simplest algorithm is the nearest neighbor, where the key is to always visit the nearest city. The genetic algorithm utilized an evolutionary process to improve the generated solution candidates, but the refined solution comes at the cost of an increased running time. In this article, we use the heuristic ant colony optimization (ACO) algorithm [31] to plan the flight path for the second stage. This method, modeled on ant behavior, is computationally feasible even for on-board path planning on a small UAV computer.

IV. SIMULATION
The flight time of the two-stage flight strategy is simulated and compared to a row-by-row flight path as well as a lawn mower flight pattern covering the entire plant area. The simulated plant is 200×200 m, and each row is two modules high. The area-covering lawn mower flight time is only affected by the plant area for a given camera and module resolution requirement, while the path optimization and single-row flight will vary with the module density of the plant. We have, therefore, simulated two different plant geometries. The first plant is representative of southern India with a high module density. The module tilt of 10°is optimized for the latitude [32], and a row distance of 3 m is found typical for the location, based on existing plants. The second simulation pictures a typical plant in northern Germany, with a higher row distance (7 m) due to a lower solar angle. The module tilt, also here optimized for latitude, is set to 40°. The resulting number of modules is 11 800 and 8200 for the simulated Indian and German plants, respectively.
For both plants, defective modules are distributed randomly at rates between 0.1% and 5%, in steps of 0.1%. Five different plants are generated for each defect rate. The ground field of view assumes a camera resolution of 640×480 pixels, and a 30% overlap between the legs in the lawn mower flights is included for robustness. For the lawn mower flight pattern, single row flight path, and the classification stage, a cell resolution of 8×8 pixels is obtained to satisfy the requirements for defect classification with some margins. The exploration stage has a cell resolution of 2×2 pixels. We assume an average velocity of 3 m/s for the classification flights and 5 m/s for the detection phase. A maneuvering time of 2 s is added for every UAV turn for all flight patterns, which, in our experience, is a realistic average maneuvering time at the relevant velocities. The resulting flight time for the three different flight patterns is compared for different defect densities.

V. EXPERIMENTAL
A plant-scale dataset was obtained by waypoint UAV flights over the PV plant Megasol in Arvika, Sweden. Megasol consists of around 4000 modules with a tilt angle of 40°. The UAV was a DJI Matrice 100 with an Optris Pi 640 thermal camera with 640×480 pixels and a field of view of 33°×25°. UAV and camera properties are listed in Table I. The thermal imagery was recorded by an Udoo onboard computer, which also streamed the images to a ground station for real-time monitoring. For this initial evaluation of the proposed concept, we use high-resolution imagery of the entire plant recorded from an altitude of 20 m to get a sufficient module resolution for defect classification. The UAV flew along each module row in the plant, following waypoints manually extracted from the map of the plant. The duration of the flight was 32 min, not counting a stop for battery replacement due to limits of the drone used. We use these data to predict performance and flight time for different scenarios, as discussed in the following.  [33], starting at 5 degrees below the temperature of healthy modules. By ignoring lower temperatures, the background is mostly removed. An object detection algorithm based on the "Binary Large Object Detection" function (blob detection) from OpenCV [34] is then run on the images. As the vegetation in the PV plant is overgrown, we targeted the detection algorithm to search for cell hotspots, which can be caused by shadowing (among other things). Selection criteria are applied to the detected blobs to reduce false detections: Only detections with a size ranging from 10% of a cell to two cells are accepted, to include both partly shadowed cells and adjacent cell hotspots. Second, the temperature difference ΔT between the object and the background must exceed 10 degrees to be counted, in accordance with IEA criteria [14]. All detections are tracked through consecutive images to avoid duplicates. The tracking algorithm is based on "Simple object tracking with OpenCV" [35], with some modifications to better fit the task: Instead of an isotropic tracking search, we use the UAV altitude, velocity, and heading to calculate the expected position of each detection in the next frame. If a detection matches a prediction, it is associated with the previous detection, otherwise it is added to the list of detected anomalies. Finally, the global positions of all objects are estimated using the logged UAV position and orientation, together with the intrinsic and extrinsic camera parameters. The data processing steps can be performed in real-time on an on-board computer.

A. Impact of Altitude on Defect Detection and Classification
Setting the UAV altitude is a tradeoff between image quality and operation time. During the high-altitude sweep at stage 1, it is crucial to ensure a preserved detectability of the module defects, while still taking advantage of the relaxed resolution requirement of the less complex task. The size of the defect and the ΔT value are the two main factors that determine the detectability of a thermal anomaly. IEC's categorization of 12 different defect types shows a general trend: Small hotspots have high ΔT values, while large hotspots tend to have a low ΔT value. The lowest ΔT values are found in open-circuit defects, which typically reach a ΔT value of 2-7 K at inspection conditions. The smallest of these defects is an open circuit substring, which results in the heating of one-third of the module for a common module design. For a cell resolution of 2×2 pixels during the exploration stage, a substring of 20 cells will cover 800 pixels. A thermal camera with a NETD value of around 70 mK will, therefore, robustly detect these relatively large but subtle defects. Cell defects on the other hand are small in size, but must have a ΔT value of at least 10 degrees to be considered a failure [33]. It is also worth noting that a cell hotspot heats up due to a reverse bias across the shaded or damaged cell, leading to the dissipation of power in the poor cell. This means that the most severe failures with the largest losses will have a higher ΔT value, and therefore an increased detectability.
The defect with the lowest detectability out of the 12 categories presented by IEC is most probably a heated module junction box, which results in a point abnormality only some degrees warmer than a healthy junction box (≥3 K). The ΔT is expected to rise due to series resistance from bad contacts in the junction box or significant reverse current in defective bypass diodes. In both cases, the ΔT is expected to increase with severity.
An effect that occurs at large imaging distances and affects defect detection is smearing due to lens imperfections. This blur effect will distribute the area of a hotspot to a larger region, resulting in a reduced ΔT value. This is illustrated in Fig. 1(a)  and (b), where the same cell hotspot is imaged from an altitude of 30 and 80 m, respectively. A cell covers 1.6% of the module area for the imaged 60-cell module, which is correctly achieved at 30 m. At 80 m, however, the area is approximately doubled, with an area of 3.3% within the yellow temperature contour. While the larger area increases the detectability, the reduced ΔT has the opposite effect. However, even with blur effects that reduce the temperature, the ΔT will still be well within the NETD of a modern camera.

B. Efficiency of Different Flight Strategies
Three different flight patterns are simulated: Single row flight, lawn mower pattern, and the proposed two-stage flight path. The latter two are illustrated in Fig. 3 for a plant with a defect density of 0.5%. Fig. 3(a) shows the lawn mower pattern and the camera field of view at an altitude low enough to perform a defect classification. The same plant is illustrated in Fig. 3(b), with the high-altitude detection flight to map out the defect locations, followed by a low-altitude classification flight revisiting the detections only. Fig. 4 shows the UAV flight times per area for different defect rates. The lawn mower pattern and single row flight are both independent of the defect density in the plant and are, therefore, constant, while the path optimization for the two different plant geometries both has an increasing flight length with increasing defect density. The two-stage flight strategy is more beneficial for plants with sparse module densities, here exemplified by a northern German plant, compared to the denser plants common in India. The simulation shows that the path optimization is more cost-effective than a lawn mower flight path for defect densities under 2% and 3% for dense and sparse plants, representatively, and even higher defect rates compared to a single row flight. A typical defect density for utility-scale plants is around 0.5 to 1% [35], so for most plants, an optimized UAV path is advantageous. If, however, the exploration stage reveals a particularly high density of hotspots, for instance, due to shadowing from surrounding vegetation, the flight procedure can be aborted and the inspection can proceed with a lawn mower flight path. Only an initial part of the exploration stage, e.g., 10%, is needed for the flight path selection if a uniform defect density is assumed.
The path planning algorithm can be developed further to improve flight time. Detections that are within one field of view of the camera can be clustered and considered as only one target, which will lead to a shorter flight path and less battery-consuming turns. A time reduction can also be achieved by a more dynamic UAV velocity with a faster transit speed between detections. This will be especially desirable in PV plants with few defects and potentially long distances between detections.  It is worth noting that a frequent O&M routine with the replacement of defective modules will keep the inspection cost lower with a two-stage flight strategy due to the maintenance of a low defect density. This benefit is not gained with a lawn mower flight pattern. Keeping the flight time low is also of importance due to the relatively short UAV battery life, rarely more than 30 min. The difference between 25 and 35 min of flight time can, therefore, make up a large difference in operation time if a battery change is required.

C. Validation by UAV Inspection of a PV Power Plant
The defect detection algorithm was run on the Megasol image set to represent the exploration phase. The image processing steps are illustrated in Fig. 2, and the detections are marked in blue on the map in Fig. 5(a). A total of 93 cell hotspots were automatically found. A manual inspection of the data identified 81 cell hotspots. Of the 93 automated detections, 69 were hotspots, 11 hotspots with ΔT just below 10 degrees, and 13 false detections. Sixty-nine of the 81 manually determined cell hotspots in the plant were detected, which makes the detection rate 85%. Two of the false detections are outside the plant boundary, caused by parked cars that created different heat patterns. False detections outside the plant area can be avoided with a plant boundary extraction that restricts the detection area.
The 32-min flight to record high-resolution data at Megasol can be taken as a baseline case representing a single-row flight strategy. A lawn mower pattern would cover the same area in 26 min with the same camera, altitude, and velocity. We first note that for this moderate-scale plant, a fast flight at 80 m with a velocity of 5 m/s for the exploration phase would require only 4 min to cover the plant area. The automatic detections in Fig. 5(a) are taken to represent the output of this first stage. The ACO algorithm is then used to calculate a near-optimal path between the defects, which can be used by the UAV flight controller during the classification stage. The resulting path is shown as a yellow line in Fig. 5(a). The red marker is the starting and landing spot for the UAV. If the classification phase is performed with a velocity of 3 m/s at 20 m altitude, the phase requires about 7 min, which gives a total inspection time of 11 min for the two-stage flight strategy. Compared with a lawn mower flight, the achieved time reduction is 58% and 66% compared to the performed single row flight.
While only detection of cell hotspots was exemplified in this plant with severe shadowing, it serves the purpose of demonstrating the suggested flight pattern. Detection algorithms can be tuned to detect other types of defects as well, which is discussed in Section VII-A.
At Megasol, 81 thermal anomalies were found, corresponding to approximately 2% of the modules. This is a somewhat higher defect density than the typical 0.5%-1% rate. We, therefore, also estimate the flight time for a lower density of defective modules. A random sample of 40 and 20 defects was drawn from the pool of detections, corresponding to a defect density of 1% and 0.5%. The new optimized paths are shown in Fig. 5(b) and (c). This gave a total inspection time of approximately 9 and 8 min, respectively. It is, thus, clear that a two-stage flight strategy can reduce the inspection time significantly.

VIII. SUMMARY AND CONCLUSION
Today's large PV plants require efficient UAV monitoring routines to achieve cost-effective inspections. We demonstrate an automatic defect detection and tracking algorithm on a UAV waypoint mission over a PV plant in Sweden. The waypoint mission is compared to a proposed two-stage autonomous flight planning strategy. In the first exploration stage, the defects are quickly mapped out from a high altitude with a predefined flight path and automatic defect detection. An ACO algorithm is then used to estimate an optimal route that revisits the detections during the classification stage. A lower altitude and speed will in this stage allow for defect classification. The two stages let us benefit from the relaxed image quality requirements for defect detection and only image modules of interest with higher image quality. A comparison between the two flight strategies shows that the two-stage flight plan potentially leads to significant savings in operation time. This is supported by a simulation of the proposed flight strategy for different plant geometries and defect densities. The simulation shows that for representative defect densities, the two-stage flight strategy is more cost-effective than a traditional lawn mower flight pattern for both PV power plants with dense and sparse module density.